Sep 10 13:26:37.320032 ip-10-0-139-172 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 13:26:37.320046 ip-10-0-139-172 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 13:26:37.320056 ip-10-0-139-172 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 13:26:37.320408 ip-10-0-139-172 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 13:26:47.453265 ip-10-0-139-172 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 13:26:47.453284 ip-10-0-139-172 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot dc9cb8a7e27543979ccf29173e634649 -- Sep 10 13:28:40.042007 ip-10-0-139-172 systemd[1]: Starting Kubernetes Kubelet... Sep 10 13:28:40.407405 ip-10-0-139-172 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:28:40.407405 ip-10-0-139-172 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 13:28:40.407405 ip-10-0-139-172 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:28:40.407405 ip-10-0-139-172 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 13:28:40.407405 ip-10-0-139-172 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:28:40.408485 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.408402 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 13:28:40.413655 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413638 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:40.413655 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413655 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413659 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413668 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413672 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413675 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413678 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413680 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413683 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413697 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413701 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413703 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413706 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413708 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413711 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413714 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413717 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413719 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413722 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413725 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413727 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:40.413732 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413730 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413732 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413735 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413738 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413740 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413743 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413746 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413749 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413752 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413754 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413757 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413759 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413762 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413764 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413767 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413772 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413775 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413777 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413780 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413782 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:40.414239 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413784 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413787 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413790 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413792 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413795 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413797 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413800 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413803 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413806 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413808 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413811 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413813 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413816 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413818 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413821 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413823 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413826 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413831 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413835 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:40.414742 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413838 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413840 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413843 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413847 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413850 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413853 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413857 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413859 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413862 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413865 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413868 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413871 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413874 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413876 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413879 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413882 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413884 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413887 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413890 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:40.415200 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413893 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413896 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413898 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413901 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413905 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413907 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413910 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413914 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413917 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413921 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413923 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413926 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413928 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413931 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413933 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413936 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413939 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413941 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413944 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:40.415858 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.413946 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414659 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414667 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414670 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414674 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414677 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414680 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414683 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414699 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414703 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414706 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414709 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414712 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414715 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414718 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414720 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414723 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414726 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414728 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414731 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:40.416563 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414733 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414736 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414738 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414741 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414744 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414746 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414749 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414751 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414754 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414757 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414759 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414762 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414764 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414767 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414769 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414772 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414774 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414778 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414780 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414783 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:40.417519 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414785 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414788 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414790 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414793 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414796 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414800 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414802 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414805 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414808 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414810 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414813 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414817 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414820 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414823 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414826 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414829 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414833 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414836 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414839 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:40.418405 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414842 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414844 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414847 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414850 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414853 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414856 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414859 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414862 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414864 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414867 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414869 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414873 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414875 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414878 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414880 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414883 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414886 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414889 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414893 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:40.418980 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414896 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414900 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414903 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414906 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414909 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414912 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414916 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414920 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414923 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414925 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414928 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414930 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414933 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414936 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414938 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414941 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414943 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414946 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414948 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414951 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:40.419494 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414954 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.414956 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416435 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416457 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416463 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416467 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416472 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416476 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416481 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416487 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416491 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416495 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416500 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416504 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416508 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416512 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416517 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416521 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416525 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416529 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:40.420058 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416534 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416538 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416542 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416547 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416551 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416557 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416561 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416565 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416569 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416573 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416578 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416582 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416585 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416589 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416593 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416599 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416603 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416608 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416639 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:40.420577 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416647 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416652 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416656 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416661 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416666 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416671 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416675 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416678 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416683 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416704 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416709 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416714 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416718 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416724 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416728 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416735 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416742 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416746 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416750 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416757 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:40.421143 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416762 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416766 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416770 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416774 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416779 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416783 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416788 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416794 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416800 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416806 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416810 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416815 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416820 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416824 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416829 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416833 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416837 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416842 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416846 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416850 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:40.421807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416855 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416861 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416866 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416870 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416874 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416878 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416883 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416887 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416892 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416895 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416900 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416904 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416908 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416912 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416916 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416920 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416925 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416929 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416935 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:40.422297 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416942 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416947 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.416952 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417062 2568 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417113 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417123 2568 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417130 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417137 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417142 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417149 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417156 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417161 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417166 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417171 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417177 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417182 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417187 2568 flags.go:64] FLAG: --cgroup-root="" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417191 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417197 2568 flags.go:64] FLAG: --client-ca-file="" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417203 2568 flags.go:64] FLAG: --cloud-provider="external" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417210 2568 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417217 2568 flags.go:64] FLAG: --cluster-domain="" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417222 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417227 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 13:28:40.422853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417232 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417237 2568 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417244 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417248 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417253 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417259 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417264 2568 flags.go:64] FLAG: --contention-profiling="false" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417268 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417273 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417279 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417283 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417290 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417295 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417300 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417305 2568 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417310 2568 flags.go:64] FLAG: --enable-server="true" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417315 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417323 2568 flags.go:64] FLAG: --event-burst="100" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417328 2568 flags.go:64] FLAG: --event-qps="50" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417333 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417337 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417342 2568 flags.go:64] FLAG: --eviction-hard="" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417348 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417353 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417357 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 13:28:40.423446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417362 2568 flags.go:64] FLAG: --eviction-soft="" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417367 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417372 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417377 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417383 2568 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417387 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417392 2568 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417397 2568 flags.go:64] FLAG: --feature-gates="" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417403 2568 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417408 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417413 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417418 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417423 2568 flags.go:64] FLAG: --healthz-port="10248" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417427 2568 flags.go:64] FLAG: --help="false" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417433 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417438 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417444 2568 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417449 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417454 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417460 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417465 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417470 2568 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417474 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 13:28:40.424125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417479 2568 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417484 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417489 2568 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417494 2568 flags.go:64] FLAG: --kube-reserved="" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417499 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417503 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417508 2568 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417513 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417517 2568 flags.go:64] FLAG: --lock-file="" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417521 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417529 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417534 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417543 2568 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417547 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417553 2568 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417558 2568 flags.go:64] FLAG: --logging-format="text" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417563 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417568 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417573 2568 flags.go:64] FLAG: --manifest-url="" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417577 2568 flags.go:64] FLAG: --manifest-url-header="" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417585 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417590 2568 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417596 2568 flags.go:64] FLAG: --max-pods="110" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417601 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417606 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 13:28:40.424788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417611 2568 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417615 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417620 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417625 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417629 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417641 2568 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417647 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417651 2568 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417656 2568 flags.go:64] FLAG: --pod-cidr="" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417661 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417669 2568 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417674 2568 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417679 2568 flags.go:64] FLAG: --pods-per-core="0" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417683 2568 flags.go:64] FLAG: --port="10250" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417705 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417711 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d79f1c8f862c2aaa" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417716 2568 flags.go:64] FLAG: --qos-reserved="" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417721 2568 flags.go:64] FLAG: --read-only-port="10255" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417728 2568 flags.go:64] FLAG: --register-node="true" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417732 2568 flags.go:64] FLAG: --register-with-taints="" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417738 2568 flags.go:64] FLAG: --registry-burst="10" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417743 2568 flags.go:64] FLAG: --registry-qps="5" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417749 2568 flags.go:64] FLAG: --reserved-cpus="" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417754 2568 flags.go:64] FLAG: --reserved-memory="" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417760 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 13:28:40.425399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417765 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417770 2568 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417774 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417779 2568 flags.go:64] FLAG: --runonce="false" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417783 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417788 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417793 2568 flags.go:64] FLAG: --seccomp-default="false" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417798 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417802 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417807 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417813 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417818 2568 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417823 2568 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417827 2568 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417832 2568 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417837 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417842 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417847 2568 flags.go:64] FLAG: --system-cgroups="" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417851 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417861 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417866 2568 flags.go:64] FLAG: --tls-cert-file="" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417870 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417878 2568 flags.go:64] FLAG: --tls-min-version="" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417882 2568 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417887 2568 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 13:28:40.426096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417894 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417899 2568 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417904 2568 flags.go:64] FLAG: --v="2" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417910 2568 flags.go:64] FLAG: --version="false" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417918 2568 flags.go:64] FLAG: --vmodule="" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417925 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.417930 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418119 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418126 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418131 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418136 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418140 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418145 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418149 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418153 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418160 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418165 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418170 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418175 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418179 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418183 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:40.426722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418187 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418191 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418196 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418200 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418204 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418209 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418213 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418217 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418221 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418226 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418230 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418236 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418240 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418244 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418248 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418254 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418259 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418264 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418269 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418273 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:40.427278 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418277 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418281 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418286 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418290 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418294 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418298 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418303 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418307 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418311 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418315 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418319 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418323 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418327 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418331 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418335 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418340 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418344 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418348 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418354 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418358 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:40.427791 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418362 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418366 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418370 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418376 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418380 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418385 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418391 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418398 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418403 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418408 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418422 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418427 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418432 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418436 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418441 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418445 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418450 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418454 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418459 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418463 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:40.428280 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418468 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418472 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418476 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418480 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418485 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418491 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418496 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418500 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418505 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418509 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418513 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418517 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418521 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418525 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418529 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418535 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418539 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418543 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418547 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:40.428807 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418553 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418557 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418561 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418565 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418570 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.418574 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.419378 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.427350 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.427365 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.428299 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.428309 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428374 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428379 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428383 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428386 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:40.429283 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428389 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428394 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428398 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428401 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428404 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428407 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428410 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428413 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428416 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428419 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428422 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428425 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428428 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428430 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428433 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428436 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428439 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428442 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428445 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428448 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:40.429668 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428451 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428453 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428456 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428459 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428462 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428466 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428469 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428472 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428474 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428477 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428480 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428483 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428485 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428488 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428491 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428494 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428497 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428500 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428503 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428505 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:40.430183 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428508 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428510 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428514 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428518 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428521 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428524 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428527 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428530 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428532 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428535 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428537 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428540 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428542 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428545 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428547 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428550 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428552 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428556 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428561 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:40.430709 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428564 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428568 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428570 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428573 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428575 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428578 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428580 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428583 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428586 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428588 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428591 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428593 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428596 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428598 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428601 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428603 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428606 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428608 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428611 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428613 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:40.431186 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428616 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428618 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428621 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428623 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428626 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428628 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428631 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428633 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428636 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428638 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428642 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428645 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428648 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428651 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428654 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:40.431722 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428656 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.428661 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428803 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428810 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428813 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428816 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428820 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428822 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428825 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428828 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428831 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428834 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428836 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428839 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428841 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:40.432089 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428844 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428847 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428849 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428851 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428854 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428856 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428859 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428861 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428864 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428866 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428869 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428871 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428874 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428877 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428880 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428883 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428887 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428890 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428893 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428895 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:40.432445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428898 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428900 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428903 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428905 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428908 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428912 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428915 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428918 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428921 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428924 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428927 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428929 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428932 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428934 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428937 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428940 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428942 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428945 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428947 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:40.433018 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428950 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428953 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428955 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428958 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428961 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428963 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428966 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428968 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428971 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428974 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428977 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428980 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428982 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428985 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428987 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428990 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428992 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428995 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.428997 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:40.433481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429000 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429002 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429005 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429007 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429010 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429013 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429015 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429018 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429020 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429023 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429026 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429028 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429031 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429033 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429036 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429039 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429042 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429046 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429049 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429060 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:40.433968 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429063 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429067 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429070 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429074 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429076 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429079 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429082 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:40.429085 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.429089 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.429202 2568 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.431607 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.432736 2568 server.go:1019] "Starting client certificate rotation" Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.432836 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:28:40.434445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.432866 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:28:40.435634 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.435621 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:28:40.441026 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.441012 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:28:40.450509 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.450486 2568 log.go:25] "Validated CRI v1 runtime API" Sep 10 13:28:40.454126 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.454108 2568 log.go:25] "Validated CRI v1 image API" Sep 10 13:28:40.455730 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.455713 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 13:28:40.458340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.458322 2568 fs.go:135] Filesystem UUIDs: map[48746e92-3922-488c-9d87-21f9db3fd869:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cbbaf6cc-46f1-41e6-bc12-e5da3d64298f:/dev/nvme0n1p3] Sep 10 13:28:40.458422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.458339 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 13:28:40.463379 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.463359 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:28:40.464452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.464317 2568 manager.go:216] Machine: {Timestamp:2026-09-10 13:28:40.461914755 +0000 UTC m=+0.320822206 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101028 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0d4430c28d43df83edd2f2ec00dc SystemUUID:ec2c0d44-30c2-8d43-df83-edd2f2ec00dc BootID:dc9cb8a7-e275-4397-9ccf-29173e634649 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8e:f8:dd:71:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8e:f8:dd:71:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:b2:9a:58:ec:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 13:28:40.464452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.464449 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 13:28:40.464548 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.464534 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 13:28:40.468015 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.467991 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 13:28:40.468150 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.468018 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 13:28:40.469516 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.469505 2568 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 13:28:40.469550 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.469518 2568 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 13:28:40.469550 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.469525 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:28:40.470294 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.470283 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:28:40.472150 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.472140 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 13:28:40.476813 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476802 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:28:40.476923 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476914 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 13:28:40.476953 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476944 2568 kubelet.go:486] "Attempting to sync node with API server" Sep 10 13:28:40.476979 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476953 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 13:28:40.476979 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476964 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 13:28:40.476979 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476974 2568 kubelet.go:398] "Adding apiserver pod source" Sep 10 13:28:40.477065 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.476981 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 13:28:40.477348 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.477335 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:28:40.477408 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.477354 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:28:40.479830 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.479815 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 13:28:40.482543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482527 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 13:28:40.482630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482565 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 13:28:40.482775 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482763 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 13:28:40.482842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482786 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 13:28:40.482842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482795 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 13:28:40.482842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482804 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 13:28:40.482842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482815 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 13:28:40.482842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.482823 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 13:28:40.483427 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.483417 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 13:28:40.483478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.483429 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 13:28:40.483478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.483439 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 13:28:40.483478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.483447 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 13:28:40.483478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.483472 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 13:28:40.483644 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.483485 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 13:28:40.484433 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.484421 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 13:28:40.484490 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.484435 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 13:28:40.486091 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.486049 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:40.486222 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.486204 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:40.488315 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.488299 2568 server.go:1268] "Started kubelet" Sep 10 13:28:40.488396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.488349 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 13:28:40.488625 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.488547 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 13:28:40.488706 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.488650 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 13:28:40.489269 ip-10-0-139-172 systemd[1]: Started Kubernetes Kubelet. Sep 10 13:28:40.489987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.489971 2568 server.go:310] "Adding debug handlers to kubelet server" Sep 10 13:28:40.490829 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.490730 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 13:28:40.495991 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.495964 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 13:28:40.496902 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.496881 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:40.498002 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.496951 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b20635c473 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.488273011 +0000 UTC m=+0.347180462,LastTimestamp:2026-09-10 13:28:40.488273011 +0000 UTC m=+0.347180462,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.498364 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.498348 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9ksbb" Sep 10 13:28:40.499726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.499707 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 13:28:40.500455 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.500440 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 13:28:40.500542 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.500527 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:40.501152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501135 2568 factory.go:55] Registering systemd factory Sep 10 13:28:40.501152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501152 2568 factory.go:223] Registration of the systemd container factory successfully Sep 10 13:28:40.501321 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501268 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 13:28:40.501420 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501329 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 13:28:40.501420 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501339 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 13:28:40.501420 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501393 2568 factory.go:153] Registering CRI-O factory Sep 10 13:28:40.501420 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501408 2568 factory.go:223] Registration of the crio container factory successfully Sep 10 13:28:40.501573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501452 2568 reconstruct.go:97] "Volume reconstruction finished" Sep 10 13:28:40.501573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501465 2568 reconciler.go:29] "Reconciler: start to sync state" Sep 10 13:28:40.501573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501456 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 13:28:40.501573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501494 2568 factory.go:103] Registering Raw factory Sep 10 13:28:40.501573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501512 2568 manager.go:1196] Started watching for new ooms in manager Sep 10 13:28:40.501858 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.501830 2568 manager.go:349] Starting recovery of all containers Sep 10 13:28:40.504090 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.504063 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:40.504202 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.504180 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 13:28:40.513487 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.513322 2568 manager.go:354] Recovery completed Sep 10 13:28:40.518425 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.518412 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.520148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.520132 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.520229 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.520164 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.520229 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.520178 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.521082 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521060 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 13:28:40.521177 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521090 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 13:28:40.521177 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521111 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:28:40.521177 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521070 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.521900 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521885 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.521994 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521939 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.521994 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.521956 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.522666 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.522581 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.523276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.523263 2568 policy_none.go:49] "None policy: Start" Sep 10 13:28:40.523337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.523285 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 13:28:40.523337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.523295 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 13:28:40.525174 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.525162 2568 policy_none.go:47] "Start" Sep 10 13:28:40.531395 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.531223 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.539934 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.539863 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.550080 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.550014 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.52190989 +0000 UTC m=+0.380817342,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552336 2568 manager.go:337] "Starting Device Plugin manager" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.552379 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552389 2568 server.go:87] "Starting device plugin registration server" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552703 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552717 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552796 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552867 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.552877 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.553437 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.553490 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.560933 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.521948342 +0000 UTC m=+0.380855794,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.577543 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.568885 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081caec4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.521962859 +0000 UTC m=+0.380870312,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.578075 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.577701 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b20acc23b0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.565236656 +0000 UTC m=+0.424144094,LastTimestamp:2026-09-10 13:28:40.565236656 +0000 UTC m=+0.424144094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.625976 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.625946 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 13:28:40.627232 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.627213 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 13:28:40.627340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.627256 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 13:28:40.627340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.627278 2568 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 13:28:40.627340 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.627315 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 13:28:40.640513 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.640484 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:40.653591 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.653576 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.654452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.654436 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.654531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.654463 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.654531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.654473 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.654531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.654497 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.663248 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.663123 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.654450865 +0000 UTC m=+0.513358315,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.671566 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.671500 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.654468204 +0000 UTC m=+0.513375655,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.671661 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.671567 2568 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.680420 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.680359 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081caec4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.654477243 +0000 UTC m=+0.513384693,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.712887 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.712862 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Sep 10 13:28:40.727949 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.727913 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal"] Sep 10 13:28:40.728023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.728002 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.728931 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.728913 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.729013 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.728943 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.729013 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.728965 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.731361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.731347 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.731468 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.731455 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.731507 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.731483 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.732065 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.732043 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.732166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.732078 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.732166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.732045 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.732166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.732090 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.732166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.732118 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.732304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.732175 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.734422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.734407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.734514 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.734433 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.735107 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.735091 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.735188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.735116 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.735188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.735130 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.740181 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.740106 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.728928997 +0000 UTC m=+0.587836451,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.751226 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.751165 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.728948029 +0000 UTC m=+0.587855544,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.757282 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.757257 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.760541 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.760525 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.762928 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.762869 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081caec4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.728971013 +0000 UTC m=+0.587878464,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.781623 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.781555 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.732060615 +0000 UTC m=+0.590968072,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.792735 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.792649 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.732084606 +0000 UTC m=+0.590992057,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.802525 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.802466 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.732098809 +0000 UTC m=+0.591006260,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.802733 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.802714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e0f7420fce4a564f27ac53bc181bbf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal\" (UID: \"4e0f7420fce4a564f27ac53bc181bbf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.802818 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.802741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0f7420fce4a564f27ac53bc181bbf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal\" (UID: \"4e0f7420fce4a564f27ac53bc181bbf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.802818 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.802794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578987157c44837be8ce53d3e7f8bf0f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-172.ec2.internal\" (UID: \"578987157c44837be8ce53d3e7f8bf0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.816057 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.815996 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081caec4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.73213099 +0000 UTC m=+0.591038448,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.827221 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.827156 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.732169337 +0000 UTC m=+0.591076788,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.835224 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.835162 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081caec4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.732179254 +0000 UTC m=+0.591086706,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.845237 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.845175 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.735103917 +0000 UTC m=+0.594011368,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.854105 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.854046 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.735122537 +0000 UTC m=+0.594029992,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.861995 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.861934 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081caec4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081caec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520183492 +0000 UTC m=+0.379090945,LastTimestamp:2026-09-10 13:28:40.735136504 +0000 UTC m=+0.594043955,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.872070 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.872052 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.872917 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.872902 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.873001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.872934 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.873001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.872948 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.873001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.872982 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.882131 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.882068 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c2685\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c2685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520148613 +0000 UTC m=+0.379056065,LastTimestamp:2026-09-10 13:28:40.872916509 +0000 UTC m=+0.731823964,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.885502 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.885482 2568 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.885502 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:40.885445 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-172.ec2.internal.18d3f8b2081c7b2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-172.ec2.internal,UID:ip-10-0-139-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:40.520170286 +0000 UTC m=+0.379077739,LastTimestamp:2026-09-10 13:28:40.872940347 +0000 UTC m=+0.731847802,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:40.903919 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.903898 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e0f7420fce4a564f27ac53bc181bbf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal\" (UID: \"4e0f7420fce4a564f27ac53bc181bbf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.903990 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.903924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0f7420fce4a564f27ac53bc181bbf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal\" (UID: \"4e0f7420fce4a564f27ac53bc181bbf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.903990 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.903941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578987157c44837be8ce53d3e7f8bf0f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-172.ec2.internal\" (UID: \"578987157c44837be8ce53d3e7f8bf0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.904081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.904002 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e0f7420fce4a564f27ac53bc181bbf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal\" (UID: \"4e0f7420fce4a564f27ac53bc181bbf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.904081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.904021 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0f7420fce4a564f27ac53bc181bbf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal\" (UID: \"4e0f7420fce4a564f27ac53bc181bbf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:40.904081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:40.904068 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578987157c44837be8ce53d3e7f8bf0f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-172.ec2.internal\" (UID: \"578987157c44837be8ce53d3e7f8bf0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" Sep 10 13:28:41.059772 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:41.059661 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:41.063070 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:41.063052 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" Sep 10 13:28:41.115536 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:41.115503 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Sep 10 13:28:43.075931 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.075892 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:43.076973 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.076957 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:43.077074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.076987 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:43.077074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.076998 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:43.077074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.077024 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:43.093404 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.093380 2568 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:43.297029 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.296995 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:43.369302 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.369268 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:43.371247 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.371222 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:43.371367 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.371263 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:43.371367 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.371274 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:43.384786 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.384749 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:43.507016 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.506985 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:43.558861 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:43.558826 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0f7420fce4a564f27ac53bc181bbf4.slice/crio-1d364554ebef5694a089af44a3bd34c477714317ccb37b4080810400aa360eef WatchSource:0}: Error finding container 1d364554ebef5694a089af44a3bd34c477714317ccb37b4080810400aa360eef: Status 404 returned error can't find the container with id 1d364554ebef5694a089af44a3bd34c477714317ccb37b4080810400aa360eef Sep 10 13:28:43.562696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.562665 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:43.564643 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.564629 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:28:43.573968 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.573890 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal.18d3f8b2bd95cbf0 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal,UID:4e0f7420fce4a564f27ac53bc181bbf4,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\",Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:43.564796912 +0000 UTC m=+1.633796855,LastTimestamp:2026-09-10 13:28:43.564796912 +0000 UTC m=+1.633796855,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:43.592723 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.592694 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:43.714857 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.714820 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Sep 10 13:28:43.894089 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.894007 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:43.895156 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.895129 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:43.895285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.895170 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:43.895285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.895185 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:43.895285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:43.895249 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:43.914648 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:43.914614 2568 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:44.297020 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:44.296986 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:44.423892 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:44.423810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" event={"ID":"4e0f7420fce4a564f27ac53bc181bbf4","Type":"ContainerStarted","Data":"1d364554ebef5694a089af44a3bd34c477714317ccb37b4080810400aa360eef"} Sep 10 13:28:44.594668 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:44.594568 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal.18d3f8b2fa64f9fe openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal,UID:4e0f7420fce4a564f27ac53bc181bbf4,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" in 1.02s (1.02s including waiting). Image size: 282837706 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:44.585007614 +0000 UTC m=+2.654007573,LastTimestamp:2026-09-10 13:28:44.585007614 +0000 UTC m=+2.654007573,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:44.851098 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:44.850910 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal.18d3f8b309a2f6e6 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal,UID:4e0f7420fce4a564f27ac53bc181bbf4,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:44.840728294 +0000 UTC m=+2.909728250,LastTimestamp:2026-09-10 13:28:44.840728294 +0000 UTC m=+2.909728250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:44.860105 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:44.860031 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal.18d3f8b30a2b1f68 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal,UID:4e0f7420fce4a564f27ac53bc181bbf4,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:44.84965156 +0000 UTC m=+2.918651502,LastTimestamp:2026-09-10 13:28:44.84965156 +0000 UTC m=+2.918651502,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:45.150911 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:45.150878 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578987157c44837be8ce53d3e7f8bf0f.slice/crio-b829c6d735449eab427017ba504c4bb9c12bd9a0151873e136ce07550646cf3e WatchSource:0}: Error finding container b829c6d735449eab427017ba504c4bb9c12bd9a0151873e136ce07550646cf3e: Status 404 returned error can't find the container with id b829c6d735449eab427017ba504c4bb9c12bd9a0151873e136ce07550646cf3e Sep 10 13:28:45.160946 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.160871 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-139-172.ec2.internal.18d3f8b31c369af3 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-139-172.ec2.internal,UID:578987157c44837be8ce53d3e7f8bf0f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\",Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:45.152393971 +0000 UTC m=+3.221393914,LastTimestamp:2026-09-10 13:28:45.152393971 +0000 UTC m=+3.221393914,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:45.283496 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.283456 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:45.298330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.298309 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:45.325017 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.324991 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Sep 10 13:28:45.428151 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.428060 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" event={"ID":"4e0f7420fce4a564f27ac53bc181bbf4","Type":"ContainerStarted","Data":"d0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64"} Sep 10 13:28:45.428151 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.428101 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:45.429193 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.429167 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" event={"ID":"578987157c44837be8ce53d3e7f8bf0f","Type":"ContainerStarted","Data":"b829c6d735449eab427017ba504c4bb9c12bd9a0151873e136ce07550646cf3e"} Sep 10 13:28:45.429445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.429429 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:45.429491 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.429459 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:45.429491 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.429471 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:45.429668 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.429650 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:45.480854 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.480830 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:45.484363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.484344 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:45.484495 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.484380 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:45.484495 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.484394 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:45.515455 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.515394 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:45.516326 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.516309 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:45.516432 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.516345 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:45.516432 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.516359 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:45.516432 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:45.516392 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:45.534011 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.533980 2568 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:45.918956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.918922 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:45.919141 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:45.918988 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:46.296426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:46.296396 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:46.430113 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:46.430087 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:46.431110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:46.431092 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:46.431188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:46.431123 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:46.431188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:46.431133 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:46.431371 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:46.431358 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:46.502076 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:46.502046 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:46.810161 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:46.810069 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-139-172.ec2.internal.18d3f8b37e7113be kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-139-172.ec2.internal,UID:578987157c44837be8ce53d3e7f8bf0f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\" in 1.647s (1.647s including waiting). Image size: 321671430 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:46.80039315 +0000 UTC m=+4.869393111,LastTimestamp:2026-09-10 13:28:46.80039315 +0000 UTC m=+4.869393111,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:46.858397 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:46.858328 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-139-172.ec2.internal.18d3f8b381575051 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-139-172.ec2.internal,UID:578987157c44837be8ce53d3e7f8bf0f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:46.849036369 +0000 UTC m=+4.918036335,LastTimestamp:2026-09-10 13:28:46.849036369 +0000 UTC m=+4.918036335,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:46.867267 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:46.867169 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-139-172.ec2.internal.18d3f8b381bc84a6 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-139-172.ec2.internal,UID:578987157c44837be8ce53d3e7f8bf0f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-139-172.ec2.internal,},FirstTimestamp:2026-09-10 13:28:46.855668902 +0000 UTC m=+4.924668856,LastTimestamp:2026-09-10 13:28:46.855668902 +0000 UTC m=+4.924668856,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-172.ec2.internal,}" Sep 10 13:28:47.297233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:47.297198 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:47.432903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:47.432870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" event={"ID":"578987157c44837be8ce53d3e7f8bf0f","Type":"ContainerStarted","Data":"b182f69b4c50daeba71b734f1d3af4a26289dc6a48ad871109a9535decfe4cf6"} Sep 10 13:28:47.433445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:47.432934 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:47.434027 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:47.434010 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:47.434118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:47.434037 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:47.434118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:47.434048 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:47.434222 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:47.434200 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:48.295700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.295665 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:48.434496 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.434470 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:48.435441 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.435419 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:48.435553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.435457 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:48.435553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.435472 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:48.435739 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:48.435724 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:48.535683 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:48.535656 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Sep 10 13:28:48.734660 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.734632 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:48.735726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.735707 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:48.735799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.735743 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:48.735799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.735758 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:48.735799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:48.735784 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:48.752744 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:48.752722 2568 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:49.296950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:49.296922 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:49.375319 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:49.375289 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:50.211236 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:50.211188 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:50.212271 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:50.212254 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:50.212391 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:50.212282 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:50.212391 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:50.212293 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:50.252406 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:50.252374 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:50.301443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:50.301418 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:50.301715 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:50.301695 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:50.653213 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:50.653186 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9ksbb" Sep 10 13:28:51.003681 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:51.003654 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:51.223448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.223401 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 13:28:51.306129 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.306054 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.324796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.324782 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.390439 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.390414 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.658060 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.657976 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:23:50 +0000 UTC" deadline="2028-03-23 14:29:01.573488542 +0000 UTC" Sep 10 13:28:51.658060 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.658003 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13441h0m9.915487999s" Sep 10 13:28:51.662644 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.662629 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.662702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:51.662650 2568 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.721215 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.721197 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.743565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.743543 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:51.825159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:51.825132 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.088363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:52.088332 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.088363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:52.088361 2568 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.343677 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:52.343585 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:52.369179 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:52.369154 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.401109 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:52.401088 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.488824 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:52.488796 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.771082 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:52.771054 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:52.771082 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:52.771081 2568 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:54.730520 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:54.730489 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:54.748662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:54.748637 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:54.807022 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:54.806998 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:54.943856 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:54.943830 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:55.066846 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.066790 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:55.066846 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.066817 2568 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-139-172.ec2.internal" not found Sep 10 13:28:55.153214 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.153174 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.154347 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.154329 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.154439 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.154359 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.154439 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.154370 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.154439 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.154398 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:55.164022 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:55.164002 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:55.164124 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.164028 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-172.ec2.internal\": node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.180562 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.180541 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.281509 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.281481 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.381720 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.381654 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.482353 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.482318 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.582852 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.582822 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.683418 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.683352 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.784112 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.784085 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.884593 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.884560 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:55.985119 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:55.985093 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.086176 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.086150 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.186651 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.186620 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.287568 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.287501 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.387602 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.387562 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.448168 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.448145 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal_4e0f7420fce4a564f27ac53bc181bbf4/setup/0.log" Sep 10 13:28:56.448293 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.448181 2568 generic.go:356] "Generic (PLEG): container finished" podID="4e0f7420fce4a564f27ac53bc181bbf4" containerID="d0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64" exitCode=1 Sep 10 13:28:56.448293 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.448210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" event={"ID":"4e0f7420fce4a564f27ac53bc181bbf4","Type":"ContainerDied","Data":"d0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64"} Sep 10 13:28:56.448376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.448335 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:56.450652 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.450636 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:56.450738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.450668 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:56.450738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:56.450683 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:56.450907 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.450893 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:56.488375 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.488333 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.589145 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.589114 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.689630 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.689577 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.789763 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.789732 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.890279 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.890206 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:56.990697 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:56.990662 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.091106 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.091072 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.191556 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.191489 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.252977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.252953 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:57.291813 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.291769 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.392217 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.392168 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.451563 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.451493 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal_4e0f7420fce4a564f27ac53bc181bbf4/setup/0.log" Sep 10 13:28:57.451563 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.451538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" event={"ID":"4e0f7420fce4a564f27ac53bc181bbf4","Type":"ContainerStarted","Data":"112b235a2c513638a2823a704f778f5cda43eb633735bcbe20c780b149cc5c62"} Sep 10 13:28:57.451727 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.451662 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:57.452492 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.452477 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:57.452565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.452505 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:57.452565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:57.452519 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:57.452764 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.452749 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-172.ec2.internal\" not found" node="ip-10-0-139-172.ec2.internal" Sep 10 13:28:57.493195 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.493161 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.593677 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.593639 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.694204 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.694168 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.794465 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.794442 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.895032 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.895003 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:57.995488 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:57.995463 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.096081 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.096009 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.196512 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.196480 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.297464 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.297443 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.398190 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.398130 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.498678 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.498655 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.599281 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.599255 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.700351 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.700272 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.801087 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.801062 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:58.901546 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:58.901517 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:59.002080 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.002053 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:59.103058 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.103023 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:59.203573 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.203544 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-172.ec2.internal\" not found" Sep 10 13:28:59.216043 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.216019 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:59.278092 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.278023 2568 apiserver.go:52] "Watching apiserver" Sep 10 13:28:59.288365 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.288345 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 13:28:59.289430 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.289412 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-2zrmg","openshift-ovn-kubernetes/ovnkube-node-bx2wf","kube-system/konnectivity-agent-h68xm","openshift-image-registry/node-ca-cw5l8","openshift-multus/multus-dnwpz","openshift-network-diagnostics/network-check-target-zwfnn","openshift-network-operator/iptables-alerter-8w67r","openshift-cluster-node-tuning-operator/tuned-fgmgw","openshift-multus/multus-additional-cni-plugins-pwdlg"] Sep 10 13:28:59.291495 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.291472 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" Sep 10 13:28:59.295721 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.295706 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.295784 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.295770 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:28:59.297488 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.297471 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 13:28:59.297651 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.297634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.299475 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.299455 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.302327 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.301326 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.302684 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.302662 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 13:28:59.302769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.302674 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 13:28:59.304476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.303286 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kqtrd\"" Sep 10 13:28:59.304476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.303680 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 13:28:59.304476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.303950 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 13:28:59.304476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304017 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 13:28:59.304476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304287 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 13:28:59.304781 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304453 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b2hqk\"" Sep 10 13:28:59.304837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304699 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 13:28:59.304916 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304893 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 13:28:59.305586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304766 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 13:28:59.305586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304968 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.305586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.304990 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 13:28:59.305586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.305405 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 13:28:59.305850 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.305682 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-497bg\"" Sep 10 13:28:59.306651 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.306576 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:28:59.306739 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.306670 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" Sep 10 13:28:59.308841 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.308822 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 13:28:59.309179 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.309157 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:28:59.309272 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.309207 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:28:59.309333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.309289 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 13:28:59.309333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.309295 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67hsw\"" Sep 10 13:28:59.309333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.309323 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 13:28:59.309813 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.309800 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 13:28:59.311793 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.311770 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.311927 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.311908 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:28:59.314142 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.314121 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:28:59.314237 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.314138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bpkml\"" Sep 10 13:28:59.314237 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.314208 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.314946 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.314931 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 13:28:59.315031 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.315017 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 13:28:59.316606 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.316582 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:28:59.316685 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.316607 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 13:28:59.316907 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.316893 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xxgx\"" Sep 10 13:28:59.317168 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.317157 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:28:59.318140 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.318119 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal"] Sep 10 13:28:59.318220 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.318160 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal"] Sep 10 13:28:59.318367 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.318354 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.320710 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.320693 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 13:28:59.320837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.320822 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 13:28:59.320976 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.320963 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-628mh\"" Sep 10 13:28:59.341407 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.341391 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zjjcx" Sep 10 13:28:59.352045 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.352022 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zjjcx" Sep 10 13:28:59.379001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.378977 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.379090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.379002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcdz\" (UniqueName: \"kubernetes.io/projected/162d0641-14f2-41f9-90fe-0c489578f4c6-kube-api-access-spcdz\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.392241 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.392215 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 13:28:59.467402 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.467336 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-172.ec2.internal" podStartSLOduration=0.467321327 podStartE2EDuration="467.321327ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:59.467253171 +0000 UTC m=+17.536253136" watchObservedRunningTime="2026-09-10 13:28:59.467321327 +0000 UTC m=+17.536321291" Sep 10 13:28:59.479459 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-kubelet\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.479566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-systemd-units\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.479566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cnibin\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.479566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.479566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-systemd\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.479566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysctl-d\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.479566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysctl-conf\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479582 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-run\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479609 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3480c0-853f-4f67-9427-2453aeecf5e7-tmp\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d96bfd07-ac79-4a08-9194-212590d011d9-iptables-alerter-script\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8a458ea-1aef-4cd9-b83e-70477e42d196-serviceca\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-system-cni-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-hostroot\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-tuned\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-k8s-cni-cncf-io\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479747 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-cni-bin\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmr6w\" (UniqueName: \"kubernetes.io/projected/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-kube-api-access-fmr6w\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479777 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b06c3f96-95c4-47d2-ba93-38a59005cacf-konnectivity-ca\") pod \"konnectivity-agent-h68xm\" (UID: \"b06c3f96-95c4-47d2-ba93-38a59005cacf\") " pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-etc-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.479816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-config\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-socket-dir-parent\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-daemon-config\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-etc-kubernetes\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479908 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-netns\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysconfig\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-cnibin\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-bin\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.479993 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-netd\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-netns\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480021 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-cni-multus\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480037 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-system-cni-dir\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovn-node-metrics-cert\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-env-overrides\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480103 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-modprobe-d\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480116 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-lib-modules\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-var-lib-kubelet\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-host\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480156 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b06c3f96-95c4-47d2-ba93-38a59005cacf-agent-certs\") pod \"konnectivity-agent-h68xm\" (UID: \"b06c3f96-95c4-47d2-ba93-38a59005cacf\") " pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8a458ea-1aef-4cd9-b83e-70477e42d196-host\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67m8\" (UniqueName: \"kubernetes.io/projected/c8a458ea-1aef-4cd9-b83e-70477e42d196-kube-api-access-r67m8\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-conf-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480233 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-node-log\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-log-socket\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5m4p\" (UniqueName: \"kubernetes.io/projected/0a3480c0-853f-4f67-9427-2453aeecf5e7-kube-api-access-t5m4p\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9bj\" (UniqueName: \"kubernetes.io/projected/1ac55583-190e-4a8f-8256-57e495e3ee98-kube-api-access-tx9bj\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480316 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480335 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcdz\" (UniqueName: \"kubernetes.io/projected/162d0641-14f2-41f9-90fe-0c489578f4c6-kube-api-access-spcdz\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.480823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-kubernetes\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-cni-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-systemd\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-sys\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480483 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-multus-certs\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d96bfd07-ac79-4a08-9194-212590d011d9-host-slash\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhsh\" (UniqueName: \"kubernetes.io/projected/d96bfd07-ac79-4a08-9194-212590d011d9-kube-api-access-8bhsh\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.480633 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-kubelet\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-slash\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.480709 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:59.980682395 +0000 UTC m=+18.049682338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac55583-190e-4a8f-8256-57e495e3ee98-cni-binary-copy\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72c76\" (UniqueName: \"kubernetes.io/projected/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-kube-api-access-72c76\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.481279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-var-lib-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.481743 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480805 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-ovn\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.481743 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.480826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-script-lib\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.493406 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.493383 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 13:28:59.497260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.497240 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcdz\" (UniqueName: \"kubernetes.io/projected/162d0641-14f2-41f9-90fe-0c489578f4c6-kube-api-access-spcdz\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.581112 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-script-lib\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.581251 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-kubelet\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.581251 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-systemd-units\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.581251 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-systemd-units\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.581251 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-kubelet\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.581251 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cnibin\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-systemd\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysctl-d\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cnibin\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysctl-conf\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581352 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-run\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581361 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-systemd\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3480c0-853f-4f67-9427-2453aeecf5e7-tmp\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d96bfd07-ac79-4a08-9194-212590d011d9-iptables-alerter-script\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8a458ea-1aef-4cd9-b83e-70477e42d196-serviceca\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-system-cni-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581455 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-hostroot\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-run\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581466 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysctl-d\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysctl-conf\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.581503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581492 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-tuned\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-k8s-cni-cncf-io\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-cni-bin\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581559 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmr6w\" (UniqueName: \"kubernetes.io/projected/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-kube-api-access-fmr6w\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581569 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-system-cni-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b06c3f96-95c4-47d2-ba93-38a59005cacf-konnectivity-ca\") pod \"konnectivity-agent-h68xm\" (UID: \"b06c3f96-95c4-47d2-ba93-38a59005cacf\") " pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-etc-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-cni-bin\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-config\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-socket-dir-parent\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581760 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-k8s-cni-cncf-io\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-daemon-config\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581793 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-etc-kubernetes\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-etc-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-script-lib\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581817 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-hostroot\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-netns\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysconfig\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.582240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-cnibin\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-bin\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-netns\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-socket-dir-parent\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581944 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-etc-kubernetes\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-netd\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-netns\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-cni-multus\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-system-cni-dir\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582100 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d96bfd07-ac79-4a08-9194-212590d011d9-iptables-alerter-script\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovn-node-metrics-cert\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582150 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-netns\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-env-overrides\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-modprobe-d\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582187 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-bin\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-lib-modules\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-var-lib-kubelet\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-host\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b06c3f96-95c4-47d2-ba93-38a59005cacf-agent-certs\") pod \"konnectivity-agent-h68xm\" (UID: \"b06c3f96-95c4-47d2-ba93-38a59005cacf\") " pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582297 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-netd\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8a458ea-1aef-4cd9-b83e-70477e42d196-host\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582317 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-config\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67m8\" (UniqueName: \"kubernetes.io/projected/c8a458ea-1aef-4cd9-b83e-70477e42d196-kube-api-access-r67m8\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b06c3f96-95c4-47d2-ba93-38a59005cacf-konnectivity-ca\") pod \"konnectivity-agent-h68xm\" (UID: \"b06c3f96-95c4-47d2-ba93-38a59005cacf\") " pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582387 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-host\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-conf-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582413 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-daemon-config\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582445 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582449 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582405 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-lib-modules\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-node-log\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582453 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-var-lib-kubelet\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.583797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582471 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-conf-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581844 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-log-socket\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.581992 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8a458ea-1aef-4cd9-b83e-70477e42d196-serviceca\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582524 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-cnibin\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-sysconfig\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582547 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-log-socket\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5m4p\" (UniqueName: \"kubernetes.io/projected/0a3480c0-853f-4f67-9427-2453aeecf5e7-kube-api-access-t5m4p\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-modprobe-d\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-node-log\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-var-lib-cni-multus\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9bj\" (UniqueName: \"kubernetes.io/projected/1ac55583-190e-4a8f-8256-57e495e3ee98-kube-api-access-tx9bj\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582591 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-system-cni-dir\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8a458ea-1aef-4cd9-b83e-70477e42d196-host\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582733 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-kubernetes\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582766 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-kubernetes\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-cni-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-systemd\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-sys\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.582927 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-env-overrides\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583004 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-systemd\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583007 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-multus-cni-dir\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-multus-certs\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d96bfd07-ac79-4a08-9194-212590d011d9-host-slash\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583053 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ac55583-190e-4a8f-8256-57e495e3ee98-host-run-multus-certs\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhsh\" (UniqueName: \"kubernetes.io/projected/d96bfd07-ac79-4a08-9194-212590d011d9-kube-api-access-8bhsh\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a3480c0-853f-4f67-9427-2453aeecf5e7-sys\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-kubelet\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-slash\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583123 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-kubelet\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d96bfd07-ac79-4a08-9194-212590d011d9-host-slash\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.584965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac55583-190e-4a8f-8256-57e495e3ee98-cni-binary-copy\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-slash\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583178 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72c76\" (UniqueName: \"kubernetes.io/projected/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-kube-api-access-72c76\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583215 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-var-lib-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-ovn\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583277 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-var-lib-openvswitch\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-ovn\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583313 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.583613 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac55583-190e-4a8f-8256-57e495e3ee98-cni-binary-copy\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.584748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovn-node-metrics-cert\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.584882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b06c3f96-95c4-47d2-ba93-38a59005cacf-agent-certs\") pod \"konnectivity-agent-h68xm\" (UID: \"b06c3f96-95c4-47d2-ba93-38a59005cacf\") " pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.585116 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3480c0-853f-4f67-9427-2453aeecf5e7-tmp\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.585524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.585252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a3480c0-853f-4f67-9427-2453aeecf5e7-etc-tuned\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.590830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.590808 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:28:59.590830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.590831 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:28:59.590987 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.590844 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:59.590987 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.590916 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:00.090897928 +0000 UTC m=+18.159897887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:59.592268 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.592245 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67m8\" (UniqueName: \"kubernetes.io/projected/c8a458ea-1aef-4cd9-b83e-70477e42d196-kube-api-access-r67m8\") pod \"node-ca-cw5l8\" (UID: \"c8a458ea-1aef-4cd9-b83e-70477e42d196\") " pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.592452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.592430 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhsh\" (UniqueName: \"kubernetes.io/projected/d96bfd07-ac79-4a08-9194-212590d011d9-kube-api-access-8bhsh\") pod \"iptables-alerter-8w67r\" (UID: \"d96bfd07-ac79-4a08-9194-212590d011d9\") " pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.592728 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.592709 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9bj\" (UniqueName: \"kubernetes.io/projected/1ac55583-190e-4a8f-8256-57e495e3ee98-kube-api-access-tx9bj\") pod \"multus-dnwpz\" (UID: \"1ac55583-190e-4a8f-8256-57e495e3ee98\") " pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.593534 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.593515 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72c76\" (UniqueName: \"kubernetes.io/projected/cabbefbe-25a5-4ab5-b815-8e0b2b6856f3-kube-api-access-72c76\") pod \"multus-additional-cni-plugins-pwdlg\" (UID: \"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3\") " pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.593628 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.593593 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmr6w\" (UniqueName: \"kubernetes.io/projected/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-kube-api-access-fmr6w\") pod \"ovnkube-node-bx2wf\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.593946 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.593929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5m4p\" (UniqueName: \"kubernetes.io/projected/0a3480c0-853f-4f67-9427-2453aeecf5e7-kube-api-access-t5m4p\") pod \"tuned-fgmgw\" (UID: \"0a3480c0-853f-4f67-9427-2453aeecf5e7\") " pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.609750 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.609731 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:28:59.616376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.616362 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:28:59.620576 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.620557 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf16dd8_197a_4eb1_9eba_b38d4abc0552.slice/crio-6410acf5009c5379cd8e05f3b08054a7dd0c6311b79049a1198d23ffdfab98b5 WatchSource:0}: Error finding container 6410acf5009c5379cd8e05f3b08054a7dd0c6311b79049a1198d23ffdfab98b5: Status 404 returned error can't find the container with id 6410acf5009c5379cd8e05f3b08054a7dd0c6311b79049a1198d23ffdfab98b5 Sep 10 13:28:59.623665 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.623646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cw5l8" Sep 10 13:28:59.625253 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.625233 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06c3f96_95c4_47d2_ba93_38a59005cacf.slice/crio-2a5083bfd82116e2f19c52b55a5aa6ad7c4e40a829604df24d094ff862c79064 WatchSource:0}: Error finding container 2a5083bfd82116e2f19c52b55a5aa6ad7c4e40a829604df24d094ff862c79064: Status 404 returned error can't find the container with id 2a5083bfd82116e2f19c52b55a5aa6ad7c4e40a829604df24d094ff862c79064 Sep 10 13:28:59.626202 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.626186 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dnwpz" Sep 10 13:28:59.632504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.632491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8w67r" Sep 10 13:28:59.637296 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.637277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" Sep 10 13:28:59.639988 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.639965 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a458ea_1aef_4cd9_b83e_70477e42d196.slice/crio-e70c7d6515592d5e42fd3293e1562a3d3e9dc158e4d1d15683ec55ce5c298e77 WatchSource:0}: Error finding container e70c7d6515592d5e42fd3293e1562a3d3e9dc158e4d1d15683ec55ce5c298e77: Status 404 returned error can't find the container with id e70c7d6515592d5e42fd3293e1562a3d3e9dc158e4d1d15683ec55ce5c298e77 Sep 10 13:28:59.641212 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.641115 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" Sep 10 13:28:59.641870 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.641846 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac55583_190e_4a8f_8256_57e495e3ee98.slice/crio-27c8a9a3ba61a405fd98850fa816bad004a6f003316ec1d63378bbc995a7d6d8 WatchSource:0}: Error finding container 27c8a9a3ba61a405fd98850fa816bad004a6f003316ec1d63378bbc995a7d6d8: Status 404 returned error can't find the container with id 27c8a9a3ba61a405fd98850fa816bad004a6f003316ec1d63378bbc995a7d6d8 Sep 10 13:28:59.645900 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.645877 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96bfd07_ac79_4a08_9194_212590d011d9.slice/crio-e27f51dbaeedcd726b8bbd5ebc392a8b27a1954befcf2f4c9efeab87c70b860a WatchSource:0}: Error finding container e27f51dbaeedcd726b8bbd5ebc392a8b27a1954befcf2f4c9efeab87c70b860a: Status 404 returned error can't find the container with id e27f51dbaeedcd726b8bbd5ebc392a8b27a1954befcf2f4c9efeab87c70b860a Sep 10 13:28:59.648552 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.648529 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3480c0_853f_4f67_9427_2453aeecf5e7.slice/crio-1c5e295b37795a9f59060c4635accaaa29e1ce880b76fc858565d7ca71387ddc WatchSource:0}: Error finding container 1c5e295b37795a9f59060c4635accaaa29e1ce880b76fc858565d7ca71387ddc: Status 404 returned error can't find the container with id 1c5e295b37795a9f59060c4635accaaa29e1ce880b76fc858565d7ca71387ddc Sep 10 13:28:59.652743 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:28:59.652724 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbefbe_25a5_4ab5_b815_8e0b2b6856f3.slice/crio-ee5d185af838ce5100758d1724683c72df043f0c54f08a028f1cd7455b2a198d WatchSource:0}: Error finding container ee5d185af838ce5100758d1724683c72df043f0c54f08a028f1cd7455b2a198d: Status 404 returned error can't find the container with id ee5d185af838ce5100758d1724683c72df043f0c54f08a028f1cd7455b2a198d Sep 10 13:28:59.683498 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.683472 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:59.984889 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:28:59.984853 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:28:59.985349 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.985036 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:59.985349 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:28:59.985110 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:00.985093622 +0000 UTC m=+19.054093565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:00.186183 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.186090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:00.186368 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.186277 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:00.186368 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.186298 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:00.186368 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.186312 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:00.186538 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.186376 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:01.18635908 +0000 UTC m=+19.255359026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:00.355683 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.355576 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:23:59 +0000 UTC" deadline="2028-03-13 01:46:23.721544798 +0000 UTC" Sep 10 13:29:00.355683 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.355631 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13188h17m23.365919292s" Sep 10 13:29:00.461992 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.461913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnwpz" event={"ID":"1ac55583-190e-4a8f-8256-57e495e3ee98","Type":"ContainerStarted","Data":"27c8a9a3ba61a405fd98850fa816bad004a6f003316ec1d63378bbc995a7d6d8"} Sep 10 13:29:00.463564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.463524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cw5l8" event={"ID":"c8a458ea-1aef-4cd9-b83e-70477e42d196","Type":"ContainerStarted","Data":"e70c7d6515592d5e42fd3293e1562a3d3e9dc158e4d1d15683ec55ce5c298e77"} Sep 10 13:29:00.467716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.467685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h68xm" event={"ID":"b06c3f96-95c4-47d2-ba93-38a59005cacf","Type":"ContainerStarted","Data":"2a5083bfd82116e2f19c52b55a5aa6ad7c4e40a829604df24d094ff862c79064"} Sep 10 13:29:00.470397 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.470370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"6410acf5009c5379cd8e05f3b08054a7dd0c6311b79049a1198d23ffdfab98b5"} Sep 10 13:29:00.476295 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.476269 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal_4e0f7420fce4a564f27ac53bc181bbf4/setup/0.log" Sep 10 13:29:00.476406 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.476311 2568 generic.go:356] "Generic (PLEG): container finished" podID="4e0f7420fce4a564f27ac53bc181bbf4" containerID="112b235a2c513638a2823a704f778f5cda43eb633735bcbe20c780b149cc5c62" exitCode=0 Sep 10 13:29:00.476406 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.476374 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" event={"ID":"4e0f7420fce4a564f27ac53bc181bbf4","Type":"ContainerDied","Data":"112b235a2c513638a2823a704f778f5cda43eb633735bcbe20c780b149cc5c62"} Sep 10 13:29:00.476510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.476417 2568 scope.go:117] "RemoveContainer" containerID="d0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64" Sep 10 13:29:00.476859 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.476839 2568 scope.go:117] "RemoveContainer" containerID="d0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64" Sep 10 13:29:00.485945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.485918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerStarted","Data":"ee5d185af838ce5100758d1724683c72df043f0c54f08a028f1cd7455b2a198d"} Sep 10 13:29:00.496936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.496906 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" event={"ID":"0a3480c0-853f-4f67-9427-2453aeecf5e7","Type":"ContainerStarted","Data":"1c5e295b37795a9f59060c4635accaaa29e1ce880b76fc858565d7ca71387ddc"} Sep 10 13:29:00.515767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.515471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8w67r" event={"ID":"d96bfd07-ac79-4a08-9194-212590d011d9","Type":"ContainerStarted","Data":"e27f51dbaeedcd726b8bbd5ebc392a8b27a1954befcf2f4c9efeab87c70b860a"} Sep 10 13:29:00.539948 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.539899 2568 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal_openshift-machine-config-operator_4e0f7420fce4a564f27ac53bc181bbf4_0 in pod sandbox 1d364554ebef5694a089af44a3bd34c477714317ccb37b4080810400aa360eef from index: no such id: 'd0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64'" containerID="d0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64" Sep 10 13:29:00.540102 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.539974 2568 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal_openshift-machine-config-operator_4e0f7420fce4a564f27ac53bc181bbf4_0 in pod sandbox 1d364554ebef5694a089af44a3bd34c477714317ccb37b4080810400aa360eef from index: no such id: 'd0f4bdf5cf9245aeb3e1d97b8d51b7956d4b04093d1ec5b988c4c5c7e0146a64'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal_openshift-machine-config-operator(4e0f7420fce4a564f27ac53bc181bbf4)\"" logger="UnhandledError" Sep 10 13:29:00.584887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.584856 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bx2wf"] Sep 10 13:29:00.992176 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:00.992140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:00.992751 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.992298 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:00.992751 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:00.992364 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:02.992345335 +0000 UTC m=+21.061345286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:01.193685 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.193647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:01.193852 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:01.193802 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:01.193852 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:01.193822 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:01.193852 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:01.193834 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:01.194013 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:01.193891 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:03.193872763 +0000 UTC m=+21.262872706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:01.356834 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.356640 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:23:59 +0000 UTC" deadline="2028-05-14 14:09:34.346979656 +0000 UTC" Sep 10 13:29:01.356834 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.356702 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14688h40m32.990282132s" Sep 10 13:29:01.418199 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.418163 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:01.418379 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:01.418300 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:01.419061 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.418718 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:01.419061 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:01.418817 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:01.524290 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.523643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" event={"ID":"4e0f7420fce4a564f27ac53bc181bbf4","Type":"ContainerStarted","Data":"3d6208e5c8d5f75bdf880b265f761d92f89f5cfd92bb4bc0901413927173663c"} Sep 10 13:29:01.542296 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:01.542145 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-172.ec2.internal" podStartSLOduration=2.542124845 podStartE2EDuration="2.542124845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:01.54174921 +0000 UTC m=+19.610749178" watchObservedRunningTime="2026-09-10 13:29:01.542124845 +0000 UTC m=+19.611124811" Sep 10 13:29:02.945239 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:02.945184 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:29:03.007248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:03.007215 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:03.007415 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.007364 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:03.007490 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.007424 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:07.007404205 +0000 UTC m=+25.076404165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:03.209363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:03.208959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:03.209363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.209108 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:03.209363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.209132 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:03.209363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.209144 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:03.209363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.209208 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:07.20919127 +0000 UTC m=+25.278191215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:03.417631 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:03.417588 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:03.417813 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.417719 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:03.418090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:03.418072 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:03.418195 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:03.418177 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:05.418240 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:05.418205 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:05.418721 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:05.418335 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:05.418721 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:05.418537 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:05.418721 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:05.418685 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:07.040775 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:07.040196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:07.040775 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.040372 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:07.040775 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.040436 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:15.040417076 +0000 UTC m=+33.109417025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:07.242181 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:07.242127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:07.242347 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.242285 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:07.242347 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.242306 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:07.242347 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.242319 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:07.242496 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.242378 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:15.24235984 +0000 UTC m=+33.311359787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:07.417454 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:07.417368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:07.417454 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:07.417417 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:07.417993 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.417533 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:07.417993 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:07.417712 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:09.417768 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:09.417726 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:09.418174 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:09.417729 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:09.418174 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:09.417839 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:09.418174 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:09.417959 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:11.418064 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:11.417982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:11.418507 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:11.417981 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:11.418507 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:11.418098 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:11.418507 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:11.418184 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:13.417887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:13.417852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:13.418334 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:13.417853 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:13.418334 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:13.417958 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:13.418334 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:13.418088 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:15.094000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:15.093968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:15.094432 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.094113 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:15.094432 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.094177 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:31.094160288 +0000 UTC m=+49.163160265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:15.295198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:15.295159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:15.295352 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.295321 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:15.295352 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.295342 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:15.295352 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.295351 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:15.295452 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.295405 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:31.295389409 +0000 UTC m=+49.364389369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:15.417381 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:15.417355 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:15.417465 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:15.417355 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:15.417506 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.417464 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:15.417552 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:15.417527 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:16.555635 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.555391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnwpz" event={"ID":"1ac55583-190e-4a8f-8256-57e495e3ee98","Type":"ContainerStarted","Data":"5975115200cf0c1bf8aa6775355a0edcdaa0980ff1e27a8b9e9961641a653dad"} Sep 10 13:29:16.556947 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.556925 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cw5l8" event={"ID":"c8a458ea-1aef-4cd9-b83e-70477e42d196","Type":"ContainerStarted","Data":"8e3bfb7df41e8f4df7eda4b80ec0c893b1233d244eb607dc4db827426f7eb8d9"} Sep 10 13:29:16.558243 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.558213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h68xm" event={"ID":"b06c3f96-95c4-47d2-ba93-38a59005cacf","Type":"ContainerStarted","Data":"6652d7de5419b644cce4d0d6eb886666f390e0fafa850cf8901339e5e338a490"} Sep 10 13:29:16.561031 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.561011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4"} Sep 10 13:29:16.561131 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.561062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"83f35832f3f67ae02cb3811a1bf7b2b19e956f7afa29427b4670f304bb1ac7c3"} Sep 10 13:29:16.561131 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.561100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"20ee8e42383b4905561c911a2c94736de281cd185c27a4506d3b4abd48971f85"} Sep 10 13:29:16.561131 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.561129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"65221f7cb6f397e08a2460e38789bd9901961aa663e54bcea2e3e88016fd4343"} Sep 10 13:29:16.561272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.561156 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"f11097f710642b7648150ba3988d136f4961012970859925885690f82bf21380"} Sep 10 13:29:16.561272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.561191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"a2b4847f659b5d48a21a96ec20b6207c738cb8dc8f07da4af90a5d7b4ce31f27"} Sep 10 13:29:16.562278 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.562259 2568 generic.go:356] "Generic (PLEG): container finished" podID="cabbefbe-25a5-4ab5-b815-8e0b2b6856f3" containerID="49071770b497481678c030fd83b490f47e3b2655d159d6547fb37d5f51452a09" exitCode=0 Sep 10 13:29:16.562360 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.562327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerDied","Data":"49071770b497481678c030fd83b490f47e3b2655d159d6547fb37d5f51452a09"} Sep 10 13:29:16.563842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.563825 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" event={"ID":"0a3480c0-853f-4f67-9427-2453aeecf5e7","Type":"ContainerStarted","Data":"4534fc8a6b68dc318f45a0ca6e1e23e3761aa184798fc5374e10b378fe51598b"} Sep 10 13:29:16.590164 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.590128 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dnwpz" podStartSLOduration=5.69896697 podStartE2EDuration="21.59011716s" totalImagesPullingTime="15.89115019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:16.573996425 +0000 UTC m=+34.642996390" watchObservedRunningTime="2026-09-10 13:29:16.59011716 +0000 UTC m=+34.659117125" Sep 10 13:29:16.609885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.609850 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fgmgw" podStartSLOduration=5.783698354 podStartE2EDuration="21.609838621s" totalImagesPullingTime="15.826140267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:16.58989532 +0000 UTC m=+34.658895284" watchObservedRunningTime="2026-09-10 13:29:16.609838621 +0000 UTC m=+34.678838583" Sep 10 13:29:16.643842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.641973 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h68xm" podStartSLOduration=9.818862941999999 podStartE2EDuration="21.641948709s" totalImagesPullingTime="11.823085767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:16.627213944 +0000 UTC m=+34.696213908" watchObservedRunningTime="2026-09-10 13:29:16.641948709 +0000 UTC m=+34.710948676" Sep 10 13:29:16.643842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.642101 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cw5l8" podStartSLOduration=5.812125373 podStartE2EDuration="21.642093446s" totalImagesPullingTime="15.829968073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:16.641732153 +0000 UTC m=+34.710732122" watchObservedRunningTime="2026-09-10 13:29:16.642093446 +0000 UTC m=+34.711093412" Sep 10 13:29:16.675899 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:16.675880 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:29:17.417873 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:17.417840 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:17.418064 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:17.417886 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:17.418064 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:17.417969 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:17.418174 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:17.418108 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:17.567762 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:17.567720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8w67r" event={"ID":"d96bfd07-ac79-4a08-9194-212590d011d9","Type":"ContainerStarted","Data":"0a9380b120ec03ddc918409a8b5c0baff0cf40c291080916b2ef70998b0d87b3"} Sep 10 13:29:17.588734 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:17.588680 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8w67r" podStartSLOduration=6.76462805 podStartE2EDuration="22.58866082s" totalImagesPullingTime="15.82403277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:17.588299087 +0000 UTC m=+35.657299054" watchObservedRunningTime="2026-09-10 13:29:17.58866082 +0000 UTC m=+35.657660782" Sep 10 13:29:18.574415 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:18.574306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b"} Sep 10 13:29:19.418363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:19.418332 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:19.418524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:19.418334 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:19.418524 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:19.418435 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:19.418649 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:19.418548 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:19.617497 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:19.617456 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:29:19.618380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:19.618363 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:29:20.597183 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:20.597160 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h68xm" Sep 10 13:29:21.417741 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.417551 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:21.418130 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.417554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:21.418130 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:21.417808 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:21.418130 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:21.417898 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:21.581966 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.581929 2568 generic.go:356] "Generic (PLEG): container finished" podID="cabbefbe-25a5-4ab5-b815-8e0b2b6856f3" containerID="fc06cc63df5b1933da9cb561774968d2810f32b174e8dc78c20a55a5f8cabdca" exitCode=0 Sep 10 13:29:21.582131 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.582020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerDied","Data":"fc06cc63df5b1933da9cb561774968d2810f32b174e8dc78c20a55a5f8cabdca"} Sep 10 13:29:21.585863 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.585838 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerStarted","Data":"269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311"} Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586114 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-controller" containerID="cri-o://a2b4847f659b5d48a21a96ec20b6207c738cb8dc8f07da4af90a5d7b4ce31f27" gracePeriod=30 Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586136 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="northd" containerID="cri-o://83f35832f3f67ae02cb3811a1bf7b2b19e956f7afa29427b4670f304bb1ac7c3" gracePeriod=30 Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586149 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-ovn-metrics" containerID="cri-o://20ee8e42383b4905561c911a2c94736de281cd185c27a4506d3b4abd48971f85" gracePeriod=30 Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586167 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-node" containerID="cri-o://65221f7cb6f397e08a2460e38789bd9901961aa663e54bcea2e3e88016fd4343" gracePeriod=30 Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586189 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" containerID="cri-o://c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" gracePeriod=30 Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586215 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-acl-logging" containerID="cri-o://f11097f710642b7648150ba3988d136f4961012970859925885690f82bf21380" gracePeriod=30 Sep 10 13:29:21.586354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.586123 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" containerID="cri-o://5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" gracePeriod=30 Sep 10 13:29:21.604013 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.603937 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovnkube-controller" containerID="cri-o://269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" gracePeriod=30 Sep 10 13:29:21.625027 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:21.624980 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podStartSLOduration=10.770486068 podStartE2EDuration="26.624964503s" totalImagesPullingTime="15.854478435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:21.624844032 +0000 UTC m=+39.693843998" watchObservedRunningTime="2026-09-10 13:29:21.624964503 +0000 UTC m=+39.693964468" Sep 10 13:29:22.593786 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.593686 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:22.594207 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.594108 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-node/0.log" Sep 10 13:29:22.594663 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.594644 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-acl-logging/0.log" Sep 10 13:29:22.595072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595059 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-controller/0.log" Sep 10 13:29:22.595115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595089 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" exitCode=0 Sep 10 13:29:22.595115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595102 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" exitCode=0 Sep 10 13:29:22.595115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595109 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="83f35832f3f67ae02cb3811a1bf7b2b19e956f7afa29427b4670f304bb1ac7c3" exitCode=0 Sep 10 13:29:22.595115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595115 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="20ee8e42383b4905561c911a2c94736de281cd185c27a4506d3b4abd48971f85" exitCode=143 Sep 10 13:29:22.595227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595121 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="65221f7cb6f397e08a2460e38789bd9901961aa663e54bcea2e3e88016fd4343" exitCode=143 Sep 10 13:29:22.595227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595126 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="f11097f710642b7648150ba3988d136f4961012970859925885690f82bf21380" exitCode=143 Sep 10 13:29:22.595227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595131 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="a2b4847f659b5d48a21a96ec20b6207c738cb8dc8f07da4af90a5d7b4ce31f27" exitCode=143 Sep 10 13:29:22.595227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b"} Sep 10 13:29:22.595227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595221 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4"} Sep 10 13:29:22.595357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595251 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"83f35832f3f67ae02cb3811a1bf7b2b19e956f7afa29427b4670f304bb1ac7c3"} Sep 10 13:29:22.595357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595280 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"20ee8e42383b4905561c911a2c94736de281cd185c27a4506d3b4abd48971f85"} Sep 10 13:29:22.595357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"65221f7cb6f397e08a2460e38789bd9901961aa663e54bcea2e3e88016fd4343"} Sep 10 13:29:22.595357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595331 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"f11097f710642b7648150ba3988d136f4961012970859925885690f82bf21380"} Sep 10 13:29:22.595463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:22.595356 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"a2b4847f659b5d48a21a96ec20b6207c738cb8dc8f07da4af90a5d7b4ce31f27"} Sep 10 13:29:23.417787 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:23.417759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:23.417948 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:23.417759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:23.417948 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:23.417867 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:23.417948 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:23.417921 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:23.598629 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:23.598580 2568 generic.go:356] "Generic (PLEG): container finished" podID="cabbefbe-25a5-4ab5-b815-8e0b2b6856f3" containerID="5ada84d70e7a655821ce0203168cbf8da1611cb6246f0ce1bb79e73ab4a053cb" exitCode=0 Sep 10 13:29:23.599078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:23.598632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerDied","Data":"5ada84d70e7a655821ce0203168cbf8da1611cb6246f0ce1bb79e73ab4a053cb"} Sep 10 13:29:25.364932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:25.364893 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:29:25.366001 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.365972 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:25.366849 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.366818 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:25.367740 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.367705 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:25.367836 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.367747 2568 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovnkube-controller" probeResult="unknown" Sep 10 13:29:25.368631 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.368593 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:25.369411 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.369387 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:25.370148 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.370129 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:25.370203 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.370155 2568 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovnkube-controller" probeResult="unknown" Sep 10 13:29:25.417985 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:25.417952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:25.418151 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:25.417947 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:25.418151 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.418064 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:25.418151 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:25.418125 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:26.607322 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:26.607287 2568 generic.go:356] "Generic (PLEG): container finished" podID="cabbefbe-25a5-4ab5-b815-8e0b2b6856f3" containerID="4a44e08d2c34af183bb81d00e578241be2e2b62eab4d44781d5115ff7d5526ac" exitCode=0 Sep 10 13:29:26.607738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:26.607329 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerDied","Data":"4a44e08d2c34af183bb81d00e578241be2e2b62eab4d44781d5115ff7d5526ac"} Sep 10 13:29:27.417483 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:27.417449 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:27.417483 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:27.417487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:27.417747 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:27.417581 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:27.417747 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:27.417704 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:29.418155 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:29.418122 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:29.418155 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:29.418152 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:29.418745 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:29.418261 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:29.418745 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:29.418391 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:31.098007 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.097983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:31.098302 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.098098 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:31.098302 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.098143 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:03.098129752 +0000 UTC m=+81.167129695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:31.298973 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.298917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:31.299149 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.299066 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:31.299149 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.299084 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:31.299149 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.299093 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:31.299149 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.299139 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:03.299126028 +0000 UTC m=+81.368125971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:31.417533 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.417503 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:31.417714 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.417615 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:31.417714 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.417641 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:31.417714 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.417698 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:31.586903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.586820 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:29:31.587056 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.586920 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:29:31.587246 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587217 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:31.587288 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587222 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:31.587554 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587532 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:31.587554 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587533 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:31.587828 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587806 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:31.587888 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587845 2568 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" probeResult="unknown" Sep 10 13:29:31.587937 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587806 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:31.587937 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:31.587918 2568 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" probeResult="unknown" Sep 10 13:29:31.619846 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.619822 2568 generic.go:356] "Generic (PLEG): container finished" podID="cabbefbe-25a5-4ab5-b815-8e0b2b6856f3" containerID="66f51f38200df775c044b108d374f3cc922efd69a45ca0fb4fa3b73fae00095a" exitCode=0 Sep 10 13:29:31.619973 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:31.619871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerDied","Data":"66f51f38200df775c044b108d374f3cc922efd69a45ca0fb4fa3b73fae00095a"} Sep 10 13:29:32.625087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:32.625051 2568 generic.go:356] "Generic (PLEG): container finished" podID="cabbefbe-25a5-4ab5-b815-8e0b2b6856f3" containerID="18f37d43a79175fe3d217d35024c4937d8be600015095a15dc04fe32b1d1b6fb" exitCode=0 Sep 10 13:29:32.625087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:32.625092 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerDied","Data":"18f37d43a79175fe3d217d35024c4937d8be600015095a15dc04fe32b1d1b6fb"} Sep 10 13:29:33.417367 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:33.417338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:33.417552 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:33.417440 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:33.417552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:33.417501 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:33.417670 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:33.417572 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:33.630889 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:33.630860 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" event={"ID":"cabbefbe-25a5-4ab5-b815-8e0b2b6856f3","Type":"ContainerStarted","Data":"5e4d466e245f7f0cd57fc8db28cab84b6866094ef30736e2b45d7d5a300affbc"} Sep 10 13:29:33.654122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:33.654082 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pwdlg" podStartSLOduration=11.044331989 podStartE2EDuration="38.654067331s" totalImagesPullingTime="27.609735342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:33.653975031 +0000 UTC m=+51.722975032" watchObservedRunningTime="2026-09-10 13:29:33.654067331 +0000 UTC m=+51.723067296" Sep 10 13:29:35.417488 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:35.417451 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:35.417888 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:35.417563 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:35.417888 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:35.417617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:35.417888 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:35.417714 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:37.418016 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:37.417977 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:37.418549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:37.417988 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:37.418549 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:37.418208 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:37.418549 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:37.418074 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:39.417943 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:39.417911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:39.418281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:39.417911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:39.418281 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:39.418022 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:39.418281 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:39.418089 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:41.417832 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:41.417795 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:41.417832 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:41.417837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:41.418225 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.417911 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:41.418225 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.418027 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:41.586764 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.586714 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:41.587019 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.586715 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:41.587086 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.587047 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:41.587136 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.587048 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:41.587368 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.587334 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:41.587368 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.587344 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:41.587517 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.587373 2568 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" probeResult="unknown" Sep 10 13:29:41.587517 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:41.587383 2568 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" probeResult="unknown" Sep 10 13:29:43.418072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:43.418033 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:43.418458 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:43.418033 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:43.418458 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:43.418157 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:43.418458 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:43.418210 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:45.418181 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:45.418149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:45.418560 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:45.418148 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:45.418560 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:45.418252 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:45.418560 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:45.418311 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:47.417624 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:47.417585 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:47.417624 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:47.417616 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:47.418115 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:47.417692 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:47.418115 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:47.417805 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:49.418159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:49.418125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:49.418531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:49.418125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:49.418531 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:49.418225 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:49.418531 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:49.418296 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:51.418057 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.418027 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:51.418440 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.418027 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:51.418440 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.418125 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:51.418440 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.418205 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:51.586331 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586288 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:51.586501 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586290 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:51.586654 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586623 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:51.586781 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586633 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:51.586947 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586918 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:51.587092 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586943 2568 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:51.587092 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586956 2568 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" probeResult="unknown" Sep 10 13:29:51.587092 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:51.586985 2568 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" probeResult="unknown" Sep 10 13:29:51.669759 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.669707 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovnkube-controller/0.log" Sep 10 13:29:51.671026 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.671006 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:51.671394 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.671380 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-node/0.log" Sep 10 13:29:51.671748 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.671731 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-acl-logging/0.log" Sep 10 13:29:51.672149 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.672134 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-controller/0.log" Sep 10 13:29:51.672192 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.672170 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" exitCode=137 Sep 10 13:29:51.672226 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:51.672208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311"} Sep 10 13:29:52.150944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.150915 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovnkube-controller/0.log" Sep 10 13:29:52.152345 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.152328 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:52.152751 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.152738 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-node/0.log" Sep 10 13:29:52.153113 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.153101 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-acl-logging/0.log" Sep 10 13:29:52.153519 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.153502 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-controller/0.log" Sep 10 13:29:52.170458 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.170442 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:29:52.221805 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221777 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-sz9xv"] Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221892 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-controller" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221902 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-controller" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221910 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-node" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221915 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-node" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221922 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221927 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221935 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="northd" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221940 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="northd" Sep 10 13:29:52.221944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221946 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovnkube-controller" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221951 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovnkube-controller" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221956 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-acl-logging" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221961 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-acl-logging" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221966 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-ovn-metrics" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221972 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-ovn-metrics" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221977 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.221982 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222009 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-acl-logging" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222015 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="northd" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222020 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="sbdb" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222025 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovnkube-controller" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222030 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-ovn-metrics" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222034 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="kube-rbac-proxy-node" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222039 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="nbdb" Sep 10 13:29:52.222198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.222044 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" containerName="ovn-controller" Sep 10 13:29:52.234413 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234390 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-kubelet\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234420 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-var-lib-openvswitch\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234448 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-script-lib\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234471 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-netd\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234483 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-kubelet" (OuterVolumeSpecName: "host-kubelet") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-kubelet". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234494 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-var-lib-openvswitch" (OuterVolumeSpecName: "var-lib-openvswitch") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "var-lib-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234502 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-config\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234522 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-netd" (OuterVolumeSpecName: "host-cni-netd") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-cni-netd". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234541 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-systemd-units\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234557 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-systemd\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234572 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-node-log\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234593 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-ovn\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234611 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-systemd-units" (OuterVolumeSpecName: "systemd-units") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "systemd-units". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234639 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-env-overrides\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234651 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-ovn" (OuterVolumeSpecName: "run-ovn") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "run-ovn". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234638 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-node-log" (OuterVolumeSpecName: "node-log") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "node-log". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234661 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-etc-openvswitch\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234683 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-etc-openvswitch" (OuterVolumeSpecName: "etc-openvswitch") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "etc-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234704 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovn-node-metrics-cert\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234733 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmr6w\" (UniqueName: \"kubernetes.io/projected/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-kube-api-access-fmr6w\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234747 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-bin\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-var-lib-cni-networks-ovn-kubernetes\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234799 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-openvswitch\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234832 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-openvswitch" (OuterVolumeSpecName: "run-openvswitch") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "run-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.234875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234843 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-config" (OuterVolumeSpecName: "ovnkube-config") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "ovnkube-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234856 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-bin" (OuterVolumeSpecName: "host-cni-bin") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-cni-bin". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234881 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-var-lib-cni-networks-ovn-kubernetes" (OuterVolumeSpecName: "host-var-lib-cni-networks-ovn-kubernetes") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-var-lib-cni-networks-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234907 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-log-socket" (OuterVolumeSpecName: "log-socket") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "log-socket". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235192 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-env-overrides" (OuterVolumeSpecName: "env-overrides") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "env-overrides". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234888 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-log-socket\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235292 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-slash\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235336 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-ovn-kubernetes\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235371 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-netns\") pod \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\" (UID: \"dcf16dd8-197a-4eb1-9eba-b38d4abc0552\") " Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235486 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-slash" (OuterVolumeSpecName: "host-slash") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-slash". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235529 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-netns" (OuterVolumeSpecName: "host-run-netns") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-run-netns". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.235590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235559 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-ovn-kubernetes" (OuterVolumeSpecName: "host-run-ovn-kubernetes") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "host-run-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.234887 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-script-lib" (OuterVolumeSpecName: "ovnkube-script-lib") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "ovnkube-script-lib". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235614 2568 reconciler_common.go:299] "Volume detached for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-bin\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235651 2568 reconciler_common.go:299] "Volume detached for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-var-lib-cni-networks-ovn-kubernetes\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235665 2568 reconciler_common.go:299] "Volume detached for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-openvswitch\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235679 2568 reconciler_common.go:299] "Volume detached for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-log-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.235690 2568 reconciler_common.go:299] "Volume detached for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-slash\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236059 2568 reconciler_common.go:299] "Volume detached for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-netns\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236077 2568 reconciler_common.go:299] "Volume detached for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-kubelet\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236100 2568 reconciler_common.go:299] "Volume detached for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-var-lib-openvswitch\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236114 2568 reconciler_common.go:299] "Volume detached for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-cni-netd\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236131 2568 reconciler_common.go:299] "Volume detached for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-config\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236143 2568 reconciler_common.go:299] "Volume detached for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-systemd-units\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236160 2568 reconciler_common.go:299] "Volume detached for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-node-log\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236172 2568 reconciler_common.go:299] "Volume detached for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-ovn\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236184 2568 reconciler_common.go:299] "Volume detached for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-env-overrides\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.236446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.236195 2568 reconciler_common.go:299] "Volume detached for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-etc-openvswitch\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.237407 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.237384 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.239579 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.239122 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovn-node-metrics-cert" (OuterVolumeSpecName: "ovn-node-metrics-cert") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "ovn-node-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:29:52.239705 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.239629 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-kube-api-access-fmr6w" (OuterVolumeSpecName: "kube-api-access-fmr6w") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "kube-api-access-fmr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:29:52.239952 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.239935 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-systemd" (OuterVolumeSpecName: "run-systemd") pod "dcf16dd8-197a-4eb1-9eba-b38d4abc0552" (UID: "dcf16dd8-197a-4eb1-9eba-b38d4abc0552"). InnerVolumeSpecName "run-systemd". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:52.336691 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-cni-bin\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.336855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336698 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-etc-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.336855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336719 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-env-overrides\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.336855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovn-node-metrics-cert\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.336855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336799 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-cni-netd\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.336855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336847 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-ovn\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-var-lib-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2r2\" (UniqueName: \"kubernetes.io/projected/a077ae2e-221e-4a68-847f-0dae1c4abf14-kube-api-access-sb2r2\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336924 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-systemd\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-log-socket\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.336983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-run-netns\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337028 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-node-log\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovnkube-config\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovnkube-script-lib\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337138 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-slash\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-kubelet\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-systemd-units\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337242 2568 reconciler_common.go:299] "Volume detached for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-host-run-ovn-kubernetes\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337254 2568 reconciler_common.go:299] "Volume detached for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovnkube-script-lib\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337263 2568 reconciler_common.go:299] "Volume detached for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-run-systemd\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337273 2568 reconciler_common.go:299] "Volume detached for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-ovn-node-metrics-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.337401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.337283 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmr6w\" (UniqueName: \"kubernetes.io/projected/dcf16dd8-197a-4eb1-9eba-b38d4abc0552-kube-api-access-fmr6w\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:29:52.437739 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-cni-netd\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-ovn\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-var-lib-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2r2\" (UniqueName: \"kubernetes.io/projected/a077ae2e-221e-4a68-847f-0dae1c4abf14-kube-api-access-sb2r2\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-systemd\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437820 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-cni-netd\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437828 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-ovn\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437849 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-var-lib-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437864 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437881 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-run-systemd\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-log-socket\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-run-netns\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-log-socket\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437999 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-run-netns\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-node-log\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438038 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.437996 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438069 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-node-log\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovnkube-config\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovnkube-script-lib\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-slash\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-kubelet\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438169 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-systemd-units\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-kubelet\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438212 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-slash\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438215 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-cni-bin\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-host-cni-bin\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-systemd-units\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-etc-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-env-overrides\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438321 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovn-node-metrics-cert\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.438545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a077ae2e-221e-4a68-847f-0dae1c4abf14-etc-openvswitch\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.439052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovnkube-config\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.439052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438680 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovnkube-script-lib\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.439052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.438690 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a077ae2e-221e-4a68-847f-0dae1c4abf14-env-overrides\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.440691 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.440676 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a077ae2e-221e-4a68-847f-0dae1c4abf14-ovn-node-metrics-cert\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.450194 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.450169 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2r2\" (UniqueName: \"kubernetes.io/projected/a077ae2e-221e-4a68-847f-0dae1c4abf14-kube-api-access-sb2r2\") pod \"ovnkube-node-sz9xv\" (UID: \"a077ae2e-221e-4a68-847f-0dae1c4abf14\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.550581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.550546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:52.586227 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:29:52.586200 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda077ae2e_221e_4a68_847f_0dae1c4abf14.slice/crio-674047e07520cfc064e018ca8218acc9535f85e3eb6196fb4cccb96d6297b245 WatchSource:0}: Error finding container 674047e07520cfc064e018ca8218acc9535f85e3eb6196fb4cccb96d6297b245: Status 404 returned error can't find the container with id 674047e07520cfc064e018ca8218acc9535f85e3eb6196fb4cccb96d6297b245 Sep 10 13:29:52.676280 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.676217 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovnkube-controller/0.log" Sep 10 13:29:52.677637 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.677621 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:52.678081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.678068 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/kube-rbac-proxy-node/0.log" Sep 10 13:29:52.678467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.678449 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-acl-logging/0.log" Sep 10 13:29:52.678900 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.678886 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx2wf_dcf16dd8-197a-4eb1-9eba-b38d4abc0552/ovn-controller/0.log" Sep 10 13:29:52.679013 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.678987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" event={"ID":"dcf16dd8-197a-4eb1-9eba-b38d4abc0552","Type":"ContainerDied","Data":"6410acf5009c5379cd8e05f3b08054a7dd0c6311b79049a1198d23ffdfab98b5"} Sep 10 13:29:52.679071 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.679035 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx2wf" Sep 10 13:29:52.679129 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.679111 2568 scope.go:117] "RemoveContainer" containerID="269cc33bbe3f5ed8f396e81e73683b06f5a7dceb1f434fac97b1e273b5310311" Sep 10 13:29:52.680751 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.680677 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"674047e07520cfc064e018ca8218acc9535f85e3eb6196fb4cccb96d6297b245"} Sep 10 13:29:52.686909 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.686889 2568 scope.go:117] "RemoveContainer" containerID="5fd2b62d12924bd109a63b9e86f40311c1af56de64c818ecc5a3d6abbfe9932b" Sep 10 13:29:52.694035 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.694017 2568 scope.go:117] "RemoveContainer" containerID="c2be28f99b4a4e5da3234c655016a683c367da884058d0ab0f3d9f123b3a40b4" Sep 10 13:29:52.699861 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.699847 2568 scope.go:117] "RemoveContainer" containerID="83f35832f3f67ae02cb3811a1bf7b2b19e956f7afa29427b4670f304bb1ac7c3" Sep 10 13:29:52.701696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.701680 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bx2wf"] Sep 10 13:29:52.705690 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.705647 2568 scope.go:117] "RemoveContainer" containerID="20ee8e42383b4905561c911a2c94736de281cd185c27a4506d3b4abd48971f85" Sep 10 13:29:52.707283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.707262 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bx2wf"] Sep 10 13:29:52.711146 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.711131 2568 scope.go:117] "RemoveContainer" containerID="65221f7cb6f397e08a2460e38789bd9901961aa663e54bcea2e3e88016fd4343" Sep 10 13:29:52.718036 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.718020 2568 scope.go:117] "RemoveContainer" containerID="f11097f710642b7648150ba3988d136f4961012970859925885690f82bf21380" Sep 10 13:29:52.726524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:52.726504 2568 scope.go:117] "RemoveContainer" containerID="a2b4847f659b5d48a21a96ec20b6207c738cb8dc8f07da4af90a5d7b4ce31f27" Sep 10 13:29:53.417864 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.417834 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:53.417864 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.417872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:53.418062 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:53.417942 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:53.418101 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:53.418069 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:53.686938 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.686897 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"6e4e547ef674f072fc76a8f7f9682701aee11841596de7813a0f69ccba462a5f"} Sep 10 13:29:53.687353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.686971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"883e826af44df3db42bb1271f0cf722c80e4e3b4f8d12e3f4f06f8d49060dfb8"} Sep 10 13:29:53.687353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.687005 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"8c2101be5c4c476ebcfe4321b5433f5907e2d41ff3290bdd66c054eedb463b23"} Sep 10 13:29:53.687353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.687032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"6bf9d3b690cb5196df607af179a33d9399fd7e60f62a8d0b35d35696d4653692"} Sep 10 13:29:53.687353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.687059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"50a18f0705b48a271b637af1fc2f10dd9c0d6beb78ab590daf63d6f996a50ce3"} Sep 10 13:29:53.687353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:53.687084 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"448900c863e5f236141d660595ccddb69702a280b500b2a10f61ce78ec8409a0"} Sep 10 13:29:54.420791 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:54.420759 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf16dd8-197a-4eb1-9eba-b38d4abc0552" path="/var/lib/kubelet/pods/dcf16dd8-197a-4eb1-9eba-b38d4abc0552/volumes" Sep 10 13:29:55.418126 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:55.418095 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:55.418577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:55.418104 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:55.418577 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:55.418191 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:55.418577 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:55.418288 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:55.693773 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:55.693689 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"6bdb02051dcb7fa921993dc67413a9138bdf5bdc93311d84686c23ddfd2b6409"} Sep 10 13:29:57.182872 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.182841 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tlrlg"] Sep 10 13:29:57.184655 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.184640 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.187270 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.187245 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 13:29:57.187270 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.187260 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glzgh\"" Sep 10 13:29:57.187384 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.187277 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 13:29:57.268202 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.268173 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-tmp-dir\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.268355 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.268222 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-hosts-file\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.268355 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.268287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588kp\" (UniqueName: \"kubernetes.io/projected/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-kube-api-access-588kp\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.369510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.369480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-tmp-dir\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.369700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.369524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-hosts-file\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.369700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.369548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588kp\" (UniqueName: \"kubernetes.io/projected/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-kube-api-access-588kp\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.369700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.369651 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-hosts-file\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.369819 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.369804 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-tmp-dir\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.378983 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.378954 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-588kp\" (UniqueName: \"kubernetes.io/projected/7f6499f9-7d26-43d2-a1b9-f86c25c9ce80-kube-api-access-588kp\") pod \"node-resolver-tlrlg\" (UID: \"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80\") " pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.418246 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.418220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:57.418359 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:57.418338 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:57.418411 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.418390 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:57.418545 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:57.418524 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:29:57.492639 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.492573 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlrlg" Sep 10 13:29:57.502360 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:29:57.501976 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6499f9_7d26_43d2_a1b9_f86c25c9ce80.slice/crio-71d61b6d441ca41805794e657bc24922379b692c5571d3c1b93352c2751a2aba WatchSource:0}: Error finding container 71d61b6d441ca41805794e657bc24922379b692c5571d3c1b93352c2751a2aba: Status 404 returned error can't find the container with id 71d61b6d441ca41805794e657bc24922379b692c5571d3c1b93352c2751a2aba Sep 10 13:29:57.701804 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.701719 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" event={"ID":"a077ae2e-221e-4a68-847f-0dae1c4abf14","Type":"ContainerStarted","Data":"a5d32f120304fe97ebc1be2ea22497993a6e9b423b86f3e63fcc7369d180fa40"} Sep 10 13:29:57.701978 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.701958 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:57.702055 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.702030 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:57.703292 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.703271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlrlg" event={"ID":"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80","Type":"ContainerStarted","Data":"b64eea3b571302236291fd4414067cc95598556cb665a122d4195e86e6cd4153"} Sep 10 13:29:57.703380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.703306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlrlg" event={"ID":"7f6499f9-7d26-43d2-a1b9-f86c25c9ce80","Type":"ContainerStarted","Data":"71d61b6d441ca41805794e657bc24922379b692c5571d3c1b93352c2751a2aba"} Sep 10 13:29:57.716990 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.716970 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:57.724610 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.724560 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" podStartSLOduration=5.724549287 podStartE2EDuration="5.724549287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:57.724245181 +0000 UTC m=+75.793245169" watchObservedRunningTime="2026-09-10 13:29:57.724549287 +0000 UTC m=+75.793549251" Sep 10 13:29:57.762283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:57.762240 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tlrlg" podStartSLOduration=0.762227343 podStartE2EDuration="762.227343ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:57.762099621 +0000 UTC m=+75.831099586" watchObservedRunningTime="2026-09-10 13:29:57.762227343 +0000 UTC m=+75.831227307" Sep 10 13:29:58.706158 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:58.706124 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:58.720429 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:58.720403 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:29:59.418244 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:59.418209 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:29:59.418424 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:59.418331 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:29:59.418424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:29:59.418388 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:29:59.418545 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:29:59.418510 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:01.417543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:01.417505 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:01.417933 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:01.417623 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:01.417933 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:01.417662 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:01.417933 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:01.417760 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:01.713614 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:01.713522 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/0.log" Sep 10 13:30:01.713614 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:01.713561 2568 generic.go:356] "Generic (PLEG): container finished" podID="1ac55583-190e-4a8f-8256-57e495e3ee98" containerID="5975115200cf0c1bf8aa6775355a0edcdaa0980ff1e27a8b9e9961641a653dad" exitCode=1 Sep 10 13:30:01.713782 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:01.713613 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnwpz" event={"ID":"1ac55583-190e-4a8f-8256-57e495e3ee98","Type":"ContainerDied","Data":"5975115200cf0c1bf8aa6775355a0edcdaa0980ff1e27a8b9e9961641a653dad"} Sep 10 13:30:01.713964 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:01.713948 2568 scope.go:117] "RemoveContainer" containerID="5975115200cf0c1bf8aa6775355a0edcdaa0980ff1e27a8b9e9961641a653dad" Sep 10 13:30:02.717761 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:02.717734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/0.log" Sep 10 13:30:02.718155 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:02.717806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnwpz" event={"ID":"1ac55583-190e-4a8f-8256-57e495e3ee98","Type":"ContainerStarted","Data":"eea68154a81af9e91597049b133169ea018a692b588566bf21dd8154e15924f6"} Sep 10 13:30:03.106778 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:03.106698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:03.106923 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.106843 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:03.106923 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.106898 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:07.106885296 +0000 UTC m=+145.175885242 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:03.308405 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:03.308367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:03.308567 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.308485 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:03.308567 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.308503 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:03.308567 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.308512 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:03.308567 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.308561 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:07.308548646 +0000 UTC m=+145.377548589 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:03.418145 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:03.418055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:03.418145 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:03.418063 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:03.418320 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.418255 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:03.418404 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:03.418381 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:05.418385 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:05.418266 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:05.418385 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:05.418271 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:05.418385 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:05.418381 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:05.418871 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:05.418448 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:07.417685 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:07.417504 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:07.418111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:07.417504 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:07.418111 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:07.417742 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:07.418111 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:07.417863 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:09.417558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:09.417521 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:09.417558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:09.417553 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:09.418030 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:09.417683 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:09.418030 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:09.417785 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:11.418002 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:11.417966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:11.418388 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:11.417966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:11.418388 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:11.418086 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:11.418388 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:11.418147 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:13.417678 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:13.417634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:13.417678 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:13.417678 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:13.418098 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:13.417737 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:13.418098 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:13.417806 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:15.417786 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:15.417742 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:15.418171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:15.417742 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:15.418171 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:15.417875 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:15.418171 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:15.417932 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:17.417550 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:17.417514 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:17.417937 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:17.417623 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:17.417937 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:17.417661 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:17.417937 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:17.417747 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:19.417556 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:19.417522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:19.418039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:19.417533 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:19.418039 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:19.417630 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:19.418039 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:19.417713 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:21.418052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:21.417997 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:21.418052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:21.418050 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:21.418552 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:21.418141 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:21.418552 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:21.418235 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:23.417944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:23.417896 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:23.418362 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:23.417896 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:23.418362 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:23.418015 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:23.418362 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:23.418114 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:25.417790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:25.417751 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:25.418185 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:25.417754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:25.418185 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:25.417846 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:25.418185 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:25.417925 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:27.417511 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:27.417476 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:27.417511 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:27.417505 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:27.417975 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:27.417615 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:27.417975 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:27.417707 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:29.418275 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:29.418234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:29.418275 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:29.418268 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:29.418732 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:29.418359 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:29.418732 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:29.418431 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:30.720914 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:30.720872 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" podUID="a077ae2e-221e-4a68-847f-0dae1c4abf14" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:30:31.418195 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:31.418160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:31.418361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:31.418166 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:31.418361 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:31.418266 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:31.418361 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:31.418342 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:33.417910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:33.417869 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:33.418302 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:33.417875 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:33.418302 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:33.417974 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:33.418302 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:33.418032 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:35.418208 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:35.418169 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:35.418696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:35.418218 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:35.418696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:35.418295 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:35.418696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:35.418398 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:37.418343 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:37.418299 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:37.418792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:37.418309 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:37.418792 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:37.418397 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:37.418792 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:37.418476 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:39.417840 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:39.417789 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:39.417840 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:39.417823 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:39.418363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:39.417910 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:39.418363 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:39.418056 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:41.417387 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:41.417355 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:41.417844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:41.417469 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:41.417844 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:41.417471 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:41.417844 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:41.417757 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:42.329484 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:42.329445 2568 kubelet_node_status.go:448] "Node not becoming ready in time after startup" Sep 10 13:30:42.364059 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:42.364021 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:43.417619 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:43.417570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:43.417619 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:43.417620 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:43.418199 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:43.417696 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:43.418199 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:43.417817 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:45.418495 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:45.418456 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:45.418893 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:45.418559 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:45.418893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:45.418617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:45.418893 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:45.418694 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:47.365297 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:47.365206 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:47.418026 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.417978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:47.418173 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.417978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:47.418173 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:47.418120 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:47.418255 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:47.418174 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:47.813281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.813254 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:30:47.813674 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.813659 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/0.log" Sep 10 13:30:47.813748 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.813695 2568 generic.go:356] "Generic (PLEG): container finished" podID="1ac55583-190e-4a8f-8256-57e495e3ee98" containerID="eea68154a81af9e91597049b133169ea018a692b588566bf21dd8154e15924f6" exitCode=1 Sep 10 13:30:47.813748 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.813730 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnwpz" event={"ID":"1ac55583-190e-4a8f-8256-57e495e3ee98","Type":"ContainerDied","Data":"eea68154a81af9e91597049b133169ea018a692b588566bf21dd8154e15924f6"} Sep 10 13:30:47.813819 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.813777 2568 scope.go:117] "RemoveContainer" containerID="5975115200cf0c1bf8aa6775355a0edcdaa0980ff1e27a8b9e9961641a653dad" Sep 10 13:30:47.814116 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:47.814095 2568 scope.go:117] "RemoveContainer" containerID="eea68154a81af9e91597049b133169ea018a692b588566bf21dd8154e15924f6" Sep 10 13:30:47.814256 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:47.814238 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-multus pod=multus-dnwpz_openshift-multus(1ac55583-190e-4a8f-8256-57e495e3ee98)\"" pod="openshift-multus/multus-dnwpz" podUID="1ac55583-190e-4a8f-8256-57e495e3ee98" Sep 10 13:30:48.816880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:48.816856 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:30:49.417713 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:49.417667 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:49.417893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:49.417667 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:49.417893 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:49.417820 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:49.417893 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:49.417863 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:51.417967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:51.417918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:51.418521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:51.417916 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:51.418521 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:51.418038 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:51.418521 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:51.418110 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:52.365563 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:52.365530 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:53.418220 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:53.418183 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:53.418702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:53.418284 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:53.418702 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:53.418349 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:53.418702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:53.418488 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:55.417745 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:55.417711 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:55.418146 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:55.417711 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:55.418146 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:55.417822 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:55.418146 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:55.417877 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:56.127554 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.127518 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ldfgt"] Sep 10 13:30:56.131480 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.131463 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.131544 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:56.131529 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:30:56.254839 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.254799 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.254839 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.254841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-dbus\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.254999 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.254906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-kubelet-config\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.355885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.355844 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.355885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.355885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-dbus\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.356075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.355932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-kubelet-config\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.356075 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:56.355990 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:56.356075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.356012 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-kubelet-config\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.356075 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:56.356052 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:56.856036602 +0000 UTC m=+134.925036545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:56.356201 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.356095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-dbus\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.858880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:56.858843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:56.859269 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:56.858974 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:56.859269 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:56.859021 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:57.859009784 +0000 UTC m=+135.928009728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:57.367068 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:57.367034 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:57.417771 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:57.417745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:57.417893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:57.417746 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:57.417893 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:57.417873 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:57.417960 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:57.417930 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:57.866048 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:57.866017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:57.866428 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:57.866133 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:57.866428 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:57.866180 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.86616854 +0000 UTC m=+137.935168482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:58.418289 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:58.418254 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:58.418448 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:58.418372 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:30:59.418066 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:59.418017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:30:59.418445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:59.418017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:30:59.418445 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:59.418180 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:30:59.418445 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:59.418202 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:30:59.880731 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:30:59.880692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:30:59.880917 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:59.880834 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:59.880917 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:30:59.880896 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.880881685 +0000 UTC m=+141.949881634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:00.417950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:00.417908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:00.418133 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:00.418052 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:00.720072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:00.720035 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" podUID="a077ae2e-221e-4a68-847f-0dae1c4abf14" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:31:01.418409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:01.418364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:01.418807 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:01.418423 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:01.418807 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:01.418487 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:01.418807 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:01.418782 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:02.367391 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:02.367345 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:02.419143 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:02.419111 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:02.419754 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:02.419230 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:02.419754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:02.419285 2568 scope.go:117] "RemoveContainer" containerID="eea68154a81af9e91597049b133169ea018a692b588566bf21dd8154e15924f6" Sep 10 13:31:02.848242 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:02.848214 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:31:02.848380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:02.848275 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnwpz" event={"ID":"1ac55583-190e-4a8f-8256-57e495e3ee98","Type":"ContainerStarted","Data":"cc1866a7e8f19a7e09274922fa933e8e905aa0b5be740e442ae5759d48814c76"} Sep 10 13:31:03.417736 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:03.417697 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:03.417908 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:03.417697 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:03.417908 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:03.417842 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:03.417908 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:03.417873 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:03.902801 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:03.902765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:03.903167 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:03.902920 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:03.903167 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:03.902978 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:11.902961797 +0000 UTC m=+149.971961748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:04.417932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:04.417893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:04.418148 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:04.418128 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:05.418166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:05.418124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:05.418565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:05.418124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:05.418565 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:05.418240 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:05.418565 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:05.418313 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:06.418245 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:06.418203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:06.418755 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:06.418337 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:07.120119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:07.120077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:07.120287 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.120209 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:31:07.120287 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.120261 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs podName:162d0641-14f2-41f9-90fe-0c489578f4c6 nodeName:}" failed. No retries permitted until 2026-09-10 13:33:09.120246629 +0000 UTC m=+267.189246572 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs") pod "network-metrics-daemon-2zrmg" (UID: "162d0641-14f2-41f9-90fe-0c489578f4c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:31:07.321700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:07.321641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:07.321874 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.321806 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:31:07.321874 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.321828 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:31:07.321874 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.321838 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2ml2w for pod openshift-network-diagnostics/network-check-target-zwfnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:31:07.321991 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.321889 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w podName:d2874ad8-f365-42de-a179-f209937bc343 nodeName:}" failed. No retries permitted until 2026-09-10 13:33:09.321876066 +0000 UTC m=+267.390876009 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2ml2w" (UniqueName: "kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w") pod "network-check-target-zwfnn" (UID: "d2874ad8-f365-42de-a179-f209937bc343") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:31:07.368683 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.368646 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:07.418069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:07.417995 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:07.418165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:07.418007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:07.418165 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.418104 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:07.418235 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:07.418183 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:08.417908 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:08.417739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:08.418279 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:08.417987 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:09.417849 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:09.417808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:09.418026 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:09.417808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:09.418026 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:09.417909 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:09.418026 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:09.417993 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:10.418000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:10.417961 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:10.418170 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:10.418067 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:11.418407 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:11.418351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:11.418805 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:11.418351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:11.418805 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:11.418509 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:11.418805 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:11.418550 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:11.948606 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:11.948572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:11.948785 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:11.948708 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:11.948785 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:11.948765 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:27.948749102 +0000 UTC m=+166.017749046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:12.368984 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:12.368950 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:12.418493 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:12.418459 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:12.418891 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:12.418543 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:13.417904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:13.417870 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:13.417904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:13.417888 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:13.418119 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:13.417969 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:13.418119 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:13.418094 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:14.417740 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.417704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:14.418120 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:14.417838 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:14.757925 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.757887 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7"] Sep 10 13:31:14.760044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.760029 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.763174 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.763156 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 13:31:14.764518 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.764498 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 13:31:14.764722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.764704 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vqf84\"" Sep 10 13:31:14.767657 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.767631 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 13:31:14.867304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867264 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-registration-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.867304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867301 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.867566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867344 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-sys-fs\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.867566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-socket-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.867566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-device-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.867566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4dm\" (UniqueName: \"kubernetes.io/projected/17d85d98-cabb-4c91-b8b7-ce76d8a76349-kube-api-access-8b4dm\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.867566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.867526 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.967919 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.967883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-device-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.967919 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.967923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4dm\" (UniqueName: \"kubernetes.io/projected/17d85d98-cabb-4c91-b8b7-ce76d8a76349-kube-api-access-8b4dm\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.967943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.967973 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-registration-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.967990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968042 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-sys-fs\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968038 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-device-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-socket-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968049 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-sys-fs\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-registration-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.968448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.968258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d85d98-cabb-4c91-b8b7-ce76d8a76349-socket-dir\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:14.978365 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:14.978335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4dm\" (UniqueName: \"kubernetes.io/projected/17d85d98-cabb-4c91-b8b7-ce76d8a76349-kube-api-access-8b4dm\") pod \"aws-ebs-csi-driver-node-ntlk7\" (UID: \"17d85d98-cabb-4c91-b8b7-ce76d8a76349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:15.069043 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:15.068914 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" Sep 10 13:31:15.077248 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:31:15.077222 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d85d98_cabb_4c91_b8b7_ce76d8a76349.slice/crio-ea7c1a62ee5e0572fdd55eaa5306322a9ecd398661f4110564027499ac795e43 WatchSource:0}: Error finding container ea7c1a62ee5e0572fdd55eaa5306322a9ecd398661f4110564027499ac795e43: Status 404 returned error can't find the container with id ea7c1a62ee5e0572fdd55eaa5306322a9ecd398661f4110564027499ac795e43 Sep 10 13:31:15.418011 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:15.417920 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:15.418349 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:15.417920 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:15.418349 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:15.418035 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:15.418349 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:15.418120 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:15.880188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:15.880154 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" event={"ID":"17d85d98-cabb-4c91-b8b7-ce76d8a76349","Type":"ContainerStarted","Data":"ea7c1a62ee5e0572fdd55eaa5306322a9ecd398661f4110564027499ac795e43"} Sep 10 13:31:16.418140 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:16.418099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:16.418734 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:16.418247 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:17.369972 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:17.369934 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:17.418354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:17.418322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:17.418825 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:17.418325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:17.418825 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:17.418424 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:17.418825 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:17.418551 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:17.886370 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:17.886332 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" event={"ID":"17d85d98-cabb-4c91-b8b7-ce76d8a76349","Type":"ContainerStarted","Data":"9305684439bff2421cc7febdc011ef7d8fcce75b14f811599f20cca3cff73aa0"} Sep 10 13:31:17.941251 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:17.941231 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 13:31:18.417908 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:18.417872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:18.418089 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:18.417988 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:18.430769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:18.430687 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:31:17.941247429Z","UUID":"79032a0a-061c-4d86-ba7b-f62e1ef39065","Handler":null,"Name":"","Endpoint":""} Sep 10 13:31:18.432176 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:18.432158 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 13:31:18.432304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:18.432190 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 13:31:18.890929 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:18.890886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" event={"ID":"17d85d98-cabb-4c91-b8b7-ce76d8a76349","Type":"ContainerStarted","Data":"e9003eedb46f93a1a79648e1f08f42b04d76a422ebb677a09a6db0217caaf322"} Sep 10 13:31:19.418122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:19.418066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:19.418122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:19.418066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:19.418384 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:19.418331 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:19.418464 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:19.418440 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:20.418467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:20.418380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:20.418854 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:20.418506 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:20.897429 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:20.897394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" event={"ID":"17d85d98-cabb-4c91-b8b7-ce76d8a76349","Type":"ContainerStarted","Data":"f281a72427003a70376d804903951f9b2c5d3886b779891d9363267eae45b88a"} Sep 10 13:31:20.913932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:20.913888 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntlk7" podStartSLOduration=2.060018165 podStartE2EDuration="6.913874472s" totalImagesPullingTime="4.853856307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:20.913743919 +0000 UTC m=+158.982743885" watchObservedRunningTime="2026-09-10 13:31:20.913874472 +0000 UTC m=+158.982874437" Sep 10 13:31:21.418159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:21.418110 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:21.418159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:21.418162 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:21.418345 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:21.418239 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:21.418411 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:21.418375 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:22.370391 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:22.370355 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:22.418113 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:22.418080 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:22.418339 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:22.418178 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:23.418279 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:23.418230 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:23.418699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:23.418230 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:23.418699 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:23.418354 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:23.418699 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:23.418420 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:24.417820 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:24.417790 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:24.418003 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:24.417900 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:25.417880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:25.417845 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:25.418271 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:25.417845 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:25.418271 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:25.417970 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:25.418271 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:25.418043 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:26.420997 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:26.420968 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:26.421358 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:26.421060 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:26.430691 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:26.430659 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" podUID="a077ae2e-221e-4a68-847f-0dae1c4abf14" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:31:27.371335 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:27.371298 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:27.418248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:27.418212 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:27.418367 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:27.418219 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:27.418367 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:27.418329 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:27.418476 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:27.418449 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:27.956201 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:27.956158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:27.956593 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:27.956310 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:27.956593 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:27.956385 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret podName:21675ea4-e8e5-4530-8b0a-397ae6ba0f92 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:59.956370904 +0000 UTC m=+198.025370847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret") pod "global-pull-secret-syncer-ldfgt" (UID: "21675ea4-e8e5-4530-8b0a-397ae6ba0f92") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:28.420392 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:28.420365 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:28.420548 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:28.420458 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:29.417720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:29.417679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:29.418158 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:29.417686 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:29.418158 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:29.417906 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:29.418158 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:29.417785 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:30.418206 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:30.418172 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:30.418587 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:30.418281 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:31.418196 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:31.418150 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:31.418385 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:31.418154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:31.418385 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:31.418263 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:31.418385 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:31.418347 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:32.371608 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:32.371557 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:32.419060 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:32.419024 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:32.419449 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:32.419112 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:33.417914 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:33.417872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:33.418098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:33.417918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:33.418098 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:33.417989 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:33.418189 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:33.418096 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:34.418409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:34.418366 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:34.418804 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:34.418506 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:35.417485 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:35.417350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:35.417485 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:35.417351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:35.417703 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:35.417502 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:35.417703 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:35.417561 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:36.417844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:36.417807 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:36.418217 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:36.417924 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:37.372366 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:37.372331 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:37.417753 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:37.417718 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:37.417922 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:37.417728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:37.417922 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:37.417843 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:37.417922 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:37.417897 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:38.418331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:38.418290 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:38.418834 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:38.418398 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:39.418333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:39.418285 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:39.418333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:39.418297 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:39.418832 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:39.418547 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:39.418832 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:39.418648 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:40.417560 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:40.417529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:40.417740 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:40.417657 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:41.417562 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:41.417518 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:41.417958 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:41.417521 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:41.417958 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:41.417667 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:41.417958 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:41.417712 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:42.372654 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:42.372614 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:42.419130 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:42.419099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:42.419495 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:42.419200 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:43.417393 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:43.417358 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:43.417574 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:43.417460 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:43.417574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:43.417517 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:43.417677 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:43.417642 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:44.418081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:44.418049 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:44.418443 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:44.418160 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:45.418090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.417842 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:45.418090 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:45.417992 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:45.418818 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.418181 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:45.418818 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:45.418303 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:45.608344 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.608297 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ldfgt"] Sep 10 13:31:45.608531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.608433 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:45.608626 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:45.608548 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:45.609237 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.609204 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zwfnn"] Sep 10 13:31:45.610213 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.610183 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2zrmg"] Sep 10 13:31:45.952716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.952693 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:45.952870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:45.952693 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:45.952870 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:45.952800 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:45.952942 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:45.952869 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:47.374100 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:47.374065 2568 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:47.417683 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:47.417637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:47.417683 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:47.417637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:47.417900 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:47.417637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:47.417900 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:47.417753 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:47.417900 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:47.417825 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:47.417900 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:47.417894 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:49.417769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:49.417722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:49.418223 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:49.417812 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:49.418223 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:49.417905 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:49.418223 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:49.417921 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:49.418223 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:49.417971 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:49.418223 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:49.418047 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:51.417476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:51.417429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:51.417885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:51.417437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:51.417885 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:51.417543 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwfnn" podUID="d2874ad8-f365-42de-a179-f209937bc343" Sep 10 13:31:51.417885 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:51.417643 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldfgt" podUID="21675ea4-e8e5-4530-8b0a-397ae6ba0f92" Sep 10 13:31:51.417885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:51.417437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:51.417885 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:31:51.417729 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2zrmg" podUID="162d0641-14f2-41f9-90fe-0c489578f4c6" Sep 10 13:31:53.418149 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.418112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:31:53.418639 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.418228 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:31:53.418639 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.418302 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:53.423665 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.423645 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 13:31:53.424697 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.424680 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdbpg\"" Sep 10 13:31:53.424791 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.424715 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 13:31:53.424791 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.424680 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:31:53.424907 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.424850 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4vqdt\"" Sep 10 13:31:53.425479 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:53.425298 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:31:56.430801 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:56.430765 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz9xv" Sep 10 13:31:59.966226 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:59.966166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:31:59.969627 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:31:59.969592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21675ea4-e8e5-4530-8b0a-397ae6ba0f92-original-pull-secret\") pod \"global-pull-secret-syncer-ldfgt\" (UID: \"21675ea4-e8e5-4530-8b0a-397ae6ba0f92\") " pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:32:00.038851 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.038817 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldfgt" Sep 10 13:32:00.125225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.125196 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-172.ec2.internal" event="NodeReady" Sep 10 13:32:00.176328 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.176291 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v"] Sep 10 13:32:00.179947 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.179921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.182475 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.182416 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-746bc7545d-7r2l4"] Sep 10 13:32:00.184002 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.183961 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cgcxz\"" Sep 10 13:32:00.185613 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.185574 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn"] Sep 10 13:32:00.185726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.185706 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.189725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.189704 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 13:32:00.189959 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.189937 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.191157 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.191135 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.195110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.195089 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx"] Sep 10 13:32:00.195231 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.195212 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.198341 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.198323 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-56g9k"] Sep 10 13:32:00.198447 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.198430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.199970 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.199949 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 13:32:00.199970 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.199967 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rtcfw\"" Sep 10 13:32:00.200111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.199982 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 13:32:00.200772 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.200752 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 13:32:00.201462 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.201442 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-m5lfk"] Sep 10 13:32:00.201581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.201569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.202766 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.202747 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 13:32:00.204531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.204512 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-flw8r"] Sep 10 13:32:00.204661 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.204645 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.208072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.207918 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64447d557d-vpzh2"] Sep 10 13:32:00.209121 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.209106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.212710 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.212673 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt"] Sep 10 13:32:00.212940 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.212924 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.216910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.216817 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v"] Sep 10 13:32:00.217099 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.216908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.217964 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.217943 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.218330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218153 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.218330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218179 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 13:32:00.218330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218208 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.218330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218155 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.218531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218506 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 13:32:00.218531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218516 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9vfqv\"" Sep 10 13:32:00.218656 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218551 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 13:32:00.218656 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218619 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 13:32:00.218912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.218888 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62"] Sep 10 13:32:00.219283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219265 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 13:32:00.219283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219289 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.219504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219339 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.219504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219290 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.219504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219376 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.219504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219479 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 13:32:00.219754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219671 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 13:32:00.219754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 13:32:00.219754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219689 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.219754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219707 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 13:32:00.219993 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.219900 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.220097 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.220080 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 13:32:00.220534 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.220516 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w2b8x\"" Sep 10 13:32:00.220681 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.220523 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 13:32:00.220923 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.220903 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9xgqt\"" Sep 10 13:32:00.221192 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.220877 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2njnn\"" Sep 10 13:32:00.221192 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.221100 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 13:32:00.221192 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.220532 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gnz47\"" Sep 10 13:32:00.222413 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.222250 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 13:32:00.222649 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.222634 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-twrv9\"" Sep 10 13:32:00.223245 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223197 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-m5lfk"] Sep 10 13:32:00.223524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223510 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-746bc7545d-7r2l4"] Sep 10 13:32:00.223703 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223672 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt"] Sep 10 13:32:00.224169 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223766 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 13:32:00.224462 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223828 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 13:32:00.224734 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223859 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.224838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.223397 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" Sep 10 13:32:00.225574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.225554 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2kfkp"] Sep 10 13:32:00.226739 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.226716 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 13:32:00.227749 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.227718 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2n77f\"" Sep 10 13:32:00.228884 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.228865 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx"] Sep 10 13:32:00.228963 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.228902 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-flw8r"] Sep 10 13:32:00.228963 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.228923 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-56g9k"] Sep 10 13:32:00.229035 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.228982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" Sep 10 13:32:00.229082 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.229043 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64447d557d-vpzh2"] Sep 10 13:32:00.229560 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.229545 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn"] Sep 10 13:32:00.230201 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.230183 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.230313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.230295 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.230381 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.230335 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8qp6v\"" Sep 10 13:32:00.230712 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.230697 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.236923 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.236903 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dqnqm\"" Sep 10 13:32:00.238020 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.237910 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2kfkp"] Sep 10 13:32:00.238395 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.238372 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 13:32:00.238669 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.238650 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 13:32:00.244227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.244205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62"] Sep 10 13:32:00.245513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.245488 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hmmz9"] Sep 10 13:32:00.248889 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.248872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.249052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.249033 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mhj7q"] Sep 10 13:32:00.251956 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.251937 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.259225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.259204 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 13:32:00.259328 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.259204 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 13:32:00.259390 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.259214 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 13:32:00.261868 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.261851 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xxdkm\"" Sep 10 13:32:00.261868 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.261859 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 13:32:00.263257 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.263240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m68qf\"" Sep 10 13:32:00.267540 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.267519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 13:32:00.268897 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.268876 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8dab21e-4671-4586-9934-9f51c1b6dcfe-nginx-conf\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.268987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.268900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpmr\" (UniqueName: \"kubernetes.io/projected/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-kube-api-access-ndpmr\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.268987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.268920 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nz8p\" (UniqueName: \"kubernetes.io/projected/90f4e27e-5d46-4dfe-bec6-38972d80368b-kube-api-access-2nz8p\") pod \"ingress-canary-mhj7q\" (UID: \"90f4e27e-5d46-4dfe-bec6-38972d80368b\") " pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.268987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.268937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/836f29b4-3ff1-47fc-ab08-28dcdff5144c-snapshots\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.268987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.268976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-stats-auth\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.269209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmmz9"] Sep 10 13:32:00.269209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab31e74a-1aef-4a39-a7ee-757e5c05e212-config\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.269209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269077 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab31e74a-1aef-4a39-a7ee-757e5c05e212-trusted-ca\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.269209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.269209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0865057-5202-4207-a761-5ec236bf4b84-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.269209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269211 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90f4e27e-5d46-4dfe-bec6-38972d80368b-cert\") pod \"ingress-canary-mhj7q\" (UID: \"90f4e27e-5d46-4dfe-bec6-38972d80368b\") " pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269295 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246361dc-74c9-4643-8740-456a725d7cb5-tmp-dir\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtfd\" (UniqueName: \"kubernetes.io/projected/395969e1-8df4-46fc-bff6-86313e25ab87-kube-api-access-jhtfd\") pod \"network-check-source-685456655f-2kfkp\" (UID: \"395969e1-8df4-46fc-bff6-86313e25ab87\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836f29b4-3ff1-47fc-ab08-28dcdff5144c-tmp\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836f29b4-3ff1-47fc-ab08-28dcdff5144c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlj9s\" (UniqueName: \"kubernetes.io/projected/ab31e74a-1aef-4a39-a7ee-757e5c05e212-kube-api-access-zlj9s\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.269463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-bound-sa-token\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269471 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8zl\" (UniqueName: \"kubernetes.io/projected/b0865057-5202-4207-a761-5ec236bf4b84-kube-api-access-lv8zl\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269501 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtt7\" (UniqueName: \"kubernetes.io/projected/246361dc-74c9-4643-8740-456a725d7cb5-kube-api-access-fxtt7\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqs4\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-kube-api-access-hbqs4\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8c54\" (UniqueName: \"kubernetes.io/projected/836f29b4-3ff1-47fc-ab08-28dcdff5144c-kube-api-access-r8c54\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-serving-cert\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269656 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-installation-pull-secrets\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767s8\" (UniqueName: \"kubernetes.io/projected/9d949756-5384-47bf-a2e0-4380b00b5e5a-kube-api-access-767s8\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw764\" (UniqueName: \"kubernetes.io/projected/f23db181-9586-4885-ab98-3195ec654d20-kube-api-access-tw764\") pod \"volume-data-source-validator-695c895bd6-67l62\" (UID: \"f23db181-9586-4885-ab98-3195ec654d20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269746 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-trusted-ca\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2l8z\" (UniqueName: \"kubernetes.io/projected/91c4e41b-40d7-4042-8d81-b62e0d488e98-kube-api-access-f2l8z\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.269816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/246361dc-74c9-4643-8740-456a725d7cb5-config-volume\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-config\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269835 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhj7q"] Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269855 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/926a37d0-25e0-4996-8995-de7bd2115b3c-ca-trust-extracted\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-image-registry-private-configuration\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-certificates\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.269987 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0865057-5202-4207-a761-5ec236bf4b84-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836f29b4-3ff1-47fc-ab08-28dcdff5144c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-default-certificate\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd69w\" (UniqueName: \"kubernetes.io/projected/62e1b7ca-1e07-4f56-96d6-a865772d029f-kube-api-access-vd69w\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/836f29b4-3ff1-47fc-ab08-28dcdff5144c-serving-cert\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270157 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91c4e41b-40d7-4042-8d81-b62e0d488e98-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.270331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.270204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab31e74a-1aef-4a39-a7ee-757e5c05e212-serving-cert\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.331780 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.331747 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ldfgt"] Sep 10 13:32:00.370681 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.370681 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90f4e27e-5d46-4dfe-bec6-38972d80368b-cert\") pod \"ingress-canary-mhj7q\" (UID: \"90f4e27e-5d46-4dfe-bec6-38972d80368b\") " pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.370926 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246361dc-74c9-4643-8740-456a725d7cb5-tmp-dir\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.370926 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtfd\" (UniqueName: \"kubernetes.io/projected/395969e1-8df4-46fc-bff6-86313e25ab87-kube-api-access-jhtfd\") pod \"network-check-source-685456655f-2kfkp\" (UID: \"395969e1-8df4-46fc-bff6-86313e25ab87\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" Sep 10 13:32:00.370926 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836f29b4-3ff1-47fc-ab08-28dcdff5144c-tmp\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.370926 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836f29b4-3ff1-47fc-ab08-28dcdff5144c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.370926 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.370806 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:32:00.370926 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.370855 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlj9s\" (UniqueName: \"kubernetes.io/projected/ab31e74a-1aef-4a39-a7ee-757e5c05e212-kube-api-access-zlj9s\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.371233 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.370877 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls podName:62e1b7ca-1e07-4f56-96d6-a865772d029f nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.87085649 +0000 UTC m=+198.939856437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nfl8v" (UID: "62e1b7ca-1e07-4f56-96d6-a865772d029f") : secret "samples-operator-tls" not found Sep 10 13:32:00.371233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-bound-sa-token\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.371233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8zl\" (UniqueName: \"kubernetes.io/projected/b0865057-5202-4207-a761-5ec236bf4b84-kube-api-access-lv8zl\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.371233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836f29b4-3ff1-47fc-ab08-28dcdff5144c-tmp\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.371233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371204 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtt7\" (UniqueName: \"kubernetes.io/projected/246361dc-74c9-4643-8740-456a725d7cb5-kube-api-access-fxtt7\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.371233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371206 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246361dc-74c9-4643-8740-456a725d7cb5-tmp-dir\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqs4\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-kube-api-access-hbqs4\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8c54\" (UniqueName: \"kubernetes.io/projected/836f29b4-3ff1-47fc-ab08-28dcdff5144c-kube-api-access-r8c54\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-serving-cert\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-installation-pull-secrets\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371401 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767s8\" (UniqueName: \"kubernetes.io/projected/9d949756-5384-47bf-a2e0-4380b00b5e5a-kube-api-access-767s8\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371427 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw764\" (UniqueName: \"kubernetes.io/projected/f23db181-9586-4885-ab98-3195ec654d20-kube-api-access-tw764\") pod \"volume-data-source-validator-695c895bd6-67l62\" (UID: \"f23db181-9586-4885-ab98-3195ec654d20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-trusted-ca\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371483 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2l8z\" (UniqueName: \"kubernetes.io/projected/91c4e41b-40d7-4042-8d81-b62e0d488e98-kube-api-access-f2l8z\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/246361dc-74c9-4643-8740-456a725d7cb5-config-volume\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.371543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-config\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/926a37d0-25e0-4996-8995-de7bd2115b3c-ca-trust-extracted\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.371727 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.371779 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls podName:246361dc-74c9-4643-8740-456a725d7cb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.871762822 +0000 UTC m=+198.940762793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls") pod "dns-default-hmmz9" (UID: "246361dc-74c9-4643-8740-456a725d7cb5") : secret "dns-default-metrics-tls" not found Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371812 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836f29b4-3ff1-47fc-ab08-28dcdff5144c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.371845 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.371881 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls podName:91c4e41b-40d7-4042-8d81-b62e0d488e98 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.871869439 +0000 UTC m=+198.940869409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q9rnx" (UID: "91c4e41b-40d7-4042-8d81-b62e0d488e98") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371933 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-image-registry-private-configuration\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371971 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/926a37d0-25e0-4996-8995-de7bd2115b3c-ca-trust-extracted\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.371989 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-certificates\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0865057-5202-4207-a761-5ec236bf4b84-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372052 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836f29b4-3ff1-47fc-ab08-28dcdff5144c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.372101 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-default-certificate\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd69w\" (UniqueName: \"kubernetes.io/projected/62e1b7ca-1e07-4f56-96d6-a865772d029f-kube-api-access-vd69w\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/836f29b4-3ff1-47fc-ab08-28dcdff5144c-serving-cert\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91c4e41b-40d7-4042-8d81-b62e0d488e98-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab31e74a-1aef-4a39-a7ee-757e5c05e212-serving-cert\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8dab21e-4671-4586-9934-9f51c1b6dcfe-nginx-conf\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpmr\" (UniqueName: \"kubernetes.io/projected/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-kube-api-access-ndpmr\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/246361dc-74c9-4643-8740-456a725d7cb5-config-volume\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nz8p\" (UniqueName: \"kubernetes.io/projected/90f4e27e-5d46-4dfe-bec6-38972d80368b-kube-api-access-2nz8p\") pod \"ingress-canary-mhj7q\" (UID: \"90f4e27e-5d46-4dfe-bec6-38972d80368b\") " pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/836f29b4-3ff1-47fc-ab08-28dcdff5144c-snapshots\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-stats-auth\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab31e74a-1aef-4a39-a7ee-757e5c05e212-config\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372477 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab31e74a-1aef-4a39-a7ee-757e5c05e212-trusted-ca\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0865057-5202-4207-a761-5ec236bf4b84-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.372821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372562 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-certificates\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372662 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.372672 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-trusted-ca\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372721 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert podName:a8dab21e-4671-4586-9934-9f51c1b6dcfe nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.872704533 +0000 UTC m=+198.941704488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-flw8r" (UID: "a8dab21e-4671-4586-9934-9f51c1b6dcfe") : secret "networking-console-plugin-cert" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372735 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372749 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746bc7545d-7r2l4: secret "image-registry-tls" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372758 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372803 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls podName:926a37d0-25e0-4996-8995-de7bd2115b3c nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.872787683 +0000 UTC m=+198.941787663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls") pod "image-registry-746bc7545d-7r2l4" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c") : secret "image-registry-tls" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.372821 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.872811395 +0000 UTC m=+198.941811343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : secret "router-metrics-certs-default" not found Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.373111 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/836f29b4-3ff1-47fc-ab08-28dcdff5144c-snapshots\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.373248 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836f29b4-3ff1-47fc-ab08-28dcdff5144c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.373696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.373413 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-config\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.374288 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.374135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-serving-cert\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.374288 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.374256 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.874241248 +0000 UTC m=+198.943241214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : configmap references non-existent config key: service-ca.crt Sep 10 13:32:00.374404 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.374378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab31e74a-1aef-4a39-a7ee-757e5c05e212-trusted-ca\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.375208 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.374627 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-image-registry-private-configuration\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.375208 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.374965 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91c4e41b-40d7-4042-8d81-b62e0d488e98-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.375208 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.375047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab31e74a-1aef-4a39-a7ee-757e5c05e212-config\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.375707 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.375664 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8dab21e-4671-4586-9934-9f51c1b6dcfe-nginx-conf\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.376574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.376267 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-installation-pull-secrets\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.376574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.376480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-stats-auth\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.376574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.376533 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90f4e27e-5d46-4dfe-bec6-38972d80368b-cert\") pod \"ingress-canary-mhj7q\" (UID: \"90f4e27e-5d46-4dfe-bec6-38972d80368b\") " pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.376841 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.376801 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0865057-5202-4207-a761-5ec236bf4b84-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.376951 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.376924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-default-certificate\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.377340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.377309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab31e74a-1aef-4a39-a7ee-757e5c05e212-serving-cert\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.377435 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.377390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/836f29b4-3ff1-47fc-ab08-28dcdff5144c-serving-cert\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.378225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.378206 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0865057-5202-4207-a761-5ec236bf4b84-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.381075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.381058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtfd\" (UniqueName: \"kubernetes.io/projected/395969e1-8df4-46fc-bff6-86313e25ab87-kube-api-access-jhtfd\") pod \"network-check-source-685456655f-2kfkp\" (UID: \"395969e1-8df4-46fc-bff6-86313e25ab87\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" Sep 10 13:32:00.382385 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.382367 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlj9s\" (UniqueName: \"kubernetes.io/projected/ab31e74a-1aef-4a39-a7ee-757e5c05e212-kube-api-access-zlj9s\") pod \"console-operator-85b454874b-56g9k\" (UID: \"ab31e74a-1aef-4a39-a7ee-757e5c05e212\") " pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.397370 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.397321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtt7\" (UniqueName: \"kubernetes.io/projected/246361dc-74c9-4643-8740-456a725d7cb5-kube-api-access-fxtt7\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.397765 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.397741 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-bound-sa-token\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.397867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.397766 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8c54\" (UniqueName: \"kubernetes.io/projected/836f29b4-3ff1-47fc-ab08-28dcdff5144c-kube-api-access-r8c54\") pod \"insights-operator-76bcd6ccf5-m5lfk\" (UID: \"836f29b4-3ff1-47fc-ab08-28dcdff5144c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.397867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.397739 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8zl\" (UniqueName: \"kubernetes.io/projected/b0865057-5202-4207-a761-5ec236bf4b84-kube-api-access-lv8zl\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7lldn\" (UID: \"b0865057-5202-4207-a761-5ec236bf4b84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.397867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.397743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw764\" (UniqueName: \"kubernetes.io/projected/f23db181-9586-4885-ab98-3195ec654d20-kube-api-access-tw764\") pod \"volume-data-source-validator-695c895bd6-67l62\" (UID: \"f23db181-9586-4885-ab98-3195ec654d20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" Sep 10 13:32:00.398038 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.397995 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2l8z\" (UniqueName: \"kubernetes.io/projected/91c4e41b-40d7-4042-8d81-b62e0d488e98-kube-api-access-f2l8z\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.398570 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.398538 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nz8p\" (UniqueName: \"kubernetes.io/projected/90f4e27e-5d46-4dfe-bec6-38972d80368b-kube-api-access-2nz8p\") pod \"ingress-canary-mhj7q\" (UID: \"90f4e27e-5d46-4dfe-bec6-38972d80368b\") " pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.398780 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.398748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd69w\" (UniqueName: \"kubernetes.io/projected/62e1b7ca-1e07-4f56-96d6-a865772d029f-kube-api-access-vd69w\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.399129 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.399111 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767s8\" (UniqueName: \"kubernetes.io/projected/9d949756-5384-47bf-a2e0-4380b00b5e5a-kube-api-access-767s8\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.399745 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.399724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpmr\" (UniqueName: \"kubernetes.io/projected/14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e-kube-api-access-ndpmr\") pod \"service-ca-operator-859f94c4b-pbltt\" (UID: \"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.399827 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.399768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqs4\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-kube-api-access-hbqs4\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.508409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.508328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" Sep 10 13:32:00.529002 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.528977 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:00.536003 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.535978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" Sep 10 13:32:00.557657 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.557583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" Sep 10 13:32:00.564556 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.563930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" Sep 10 13:32:00.570215 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.570187 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" Sep 10 13:32:00.582254 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.581574 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhj7q" Sep 10 13:32:00.819112 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.819008 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn"] Sep 10 13:32:00.821395 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.821373 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-56g9k"] Sep 10 13:32:00.821971 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.821941 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0865057_5202_4207_a761_5ec236bf4b84.slice/crio-bce4b260a2801e37b7fcb98bd031c1d6ecc1a90a8041d447548ff84763837589 WatchSource:0}: Error finding container bce4b260a2801e37b7fcb98bd031c1d6ecc1a90a8041d447548ff84763837589: Status 404 returned error can't find the container with id bce4b260a2801e37b7fcb98bd031c1d6ecc1a90a8041d447548ff84763837589 Sep 10 13:32:00.823441 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.823416 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab31e74a_1aef_4a39_a7ee_757e5c05e212.slice/crio-ea43440ef2ae7047870f7e7fd399907acefbfa8759723f20c1ac499625831f70 WatchSource:0}: Error finding container ea43440ef2ae7047870f7e7fd399907acefbfa8759723f20c1ac499625831f70: Status 404 returned error can't find the container with id ea43440ef2ae7047870f7e7fd399907acefbfa8759723f20c1ac499625831f70 Sep 10 13:32:00.842402 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.842374 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-m5lfk"] Sep 10 13:32:00.845036 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.845006 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836f29b4_3ff1_47fc_ab08_28dcdff5144c.slice/crio-0c46537eb6ff07631e7ec2c540aae5427f3a0feaf2422e52a560a0293c74a6a5 WatchSource:0}: Error finding container 0c46537eb6ff07631e7ec2c540aae5427f3a0feaf2422e52a560a0293c74a6a5: Status 404 returned error can't find the container with id 0c46537eb6ff07631e7ec2c540aae5427f3a0feaf2422e52a560a0293c74a6a5 Sep 10 13:32:00.865357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.865314 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62"] Sep 10 13:32:00.866719 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.866689 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt"] Sep 10 13:32:00.868592 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.868568 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23db181_9586_4885_ab98_3195ec654d20.slice/crio-6fa6586d861702c804d7cea7c1ab80a19ec1fe122d88a18a127aedb6f5da9aa3 WatchSource:0}: Error finding container 6fa6586d861702c804d7cea7c1ab80a19ec1fe122d88a18a127aedb6f5da9aa3: Status 404 returned error can't find the container with id 6fa6586d861702c804d7cea7c1ab80a19ec1fe122d88a18a127aedb6f5da9aa3 Sep 10 13:32:00.869073 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.869057 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bfda3c_9cfd_4563_9e9c_a0fccb15cc9e.slice/crio-3ca1d31e078a4c1f1e048a8c08349893bf13f8816369e31f1c0bfcf609da7c6e WatchSource:0}: Error finding container 3ca1d31e078a4c1f1e048a8c08349893bf13f8816369e31f1c0bfcf609da7c6e: Status 404 returned error can't find the container with id 3ca1d31e078a4c1f1e048a8c08349893bf13f8816369e31f1c0bfcf609da7c6e Sep 10 13:32:00.890473 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:00.890652 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:00.890652 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:00.890652 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890587 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:00.890830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890655 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:32:00.890830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890666 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746bc7545d-7r2l4: secret "image-registry-tls" not found Sep 10 13:32:00.890830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890669 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls podName:91c4e41b-40d7-4042-8d81-b62e0d488e98 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.890649157 +0000 UTC m=+199.959649114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q9rnx" (UID: "91c4e41b-40d7-4042-8d81-b62e0d488e98") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:00.890830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890697 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls podName:926a37d0-25e0-4996-8995-de7bd2115b3c nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.890687851 +0000 UTC m=+199.959687795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls") pod "image-registry-746bc7545d-7r2l4" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c") : secret "image-registry-tls" not found Sep 10 13:32:00.890830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890763 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:32:00.890830 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.890821 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls podName:246361dc-74c9-4643-8740-456a725d7cb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.890803214 +0000 UTC m=+199.959803162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls") pod "dns-default-hmmz9" (UID: "246361dc-74c9-4643-8740-456a725d7cb5") : secret "dns-default-metrics-tls" not found Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.890950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891010 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891033 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.891019797 +0000 UTC m=+199.960019741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : configmap references non-existent config key: service-ca.crt Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891050 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891051 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891064 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.891047569 +0000 UTC m=+199.960047536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : secret "router-metrics-certs-default" not found Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891088 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls podName:62e1b7ca-1e07-4f56-96d6-a865772d029f nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.891076548 +0000 UTC m=+199.960076494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nfl8v" (UID: "62e1b7ca-1e07-4f56-96d6-a865772d029f") : secret "samples-operator-tls" not found Sep 10 13:32:00.891140 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:00.891103 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert podName:a8dab21e-4671-4586-9934-9f51c1b6dcfe nodeName:}" failed. No retries permitted until 2026-09-10 13:32:01.891096827 +0000 UTC m=+199.960096770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-flw8r" (UID: "a8dab21e-4671-4586-9934-9f51c1b6dcfe") : secret "networking-console-plugin-cert" not found Sep 10 13:32:00.908843 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.908800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhj7q"] Sep 10 13:32:00.910436 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.910344 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2kfkp"] Sep 10 13:32:00.912880 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.912857 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f4e27e_5d46_4dfe_bec6_38972d80368b.slice/crio-1285ace0f27e318ae891ce40870834e11b09d7595bb553e61bb44b8f1885dcd2 WatchSource:0}: Error finding container 1285ace0f27e318ae891ce40870834e11b09d7595bb553e61bb44b8f1885dcd2: Status 404 returned error can't find the container with id 1285ace0f27e318ae891ce40870834e11b09d7595bb553e61bb44b8f1885dcd2 Sep 10 13:32:00.913545 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:00.913515 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395969e1_8df4_46fc_bff6_86313e25ab87.slice/crio-ec0fe4fa8d1f70bec2cd01fdf6b3058288b2a0f550d5561bbbae485a33b368d2 WatchSource:0}: Error finding container ec0fe4fa8d1f70bec2cd01fdf6b3058288b2a0f550d5561bbbae485a33b368d2: Status 404 returned error can't find the container with id ec0fe4fa8d1f70bec2cd01fdf6b3058288b2a0f550d5561bbbae485a33b368d2 Sep 10 13:32:00.988221 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.988183 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" event={"ID":"395969e1-8df4-46fc-bff6-86313e25ab87","Type":"ContainerStarted","Data":"ec0fe4fa8d1f70bec2cd01fdf6b3058288b2a0f550d5561bbbae485a33b368d2"} Sep 10 13:32:00.989659 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.989628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-56g9k" event={"ID":"ab31e74a-1aef-4a39-a7ee-757e5c05e212","Type":"ContainerStarted","Data":"ea43440ef2ae7047870f7e7fd399907acefbfa8759723f20c1ac499625831f70"} Sep 10 13:32:00.991053 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.991017 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" event={"ID":"b0865057-5202-4207-a761-5ec236bf4b84","Type":"ContainerStarted","Data":"bce4b260a2801e37b7fcb98bd031c1d6ecc1a90a8041d447548ff84763837589"} Sep 10 13:32:00.993062 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.993027 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" event={"ID":"836f29b4-3ff1-47fc-ab08-28dcdff5144c","Type":"ContainerStarted","Data":"0c46537eb6ff07631e7ec2c540aae5427f3a0feaf2422e52a560a0293c74a6a5"} Sep 10 13:32:00.994350 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.994324 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ldfgt" event={"ID":"21675ea4-e8e5-4530-8b0a-397ae6ba0f92","Type":"ContainerStarted","Data":"2d141624ee14fa85cf6ebd14e4d10fde51b1b441bc6ce15b909291100122229a"} Sep 10 13:32:00.995604 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.995581 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhj7q" event={"ID":"90f4e27e-5d46-4dfe-bec6-38972d80368b","Type":"ContainerStarted","Data":"1285ace0f27e318ae891ce40870834e11b09d7595bb553e61bb44b8f1885dcd2"} Sep 10 13:32:00.996842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.996816 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" event={"ID":"f23db181-9586-4885-ab98-3195ec654d20","Type":"ContainerStarted","Data":"6fa6586d861702c804d7cea7c1ab80a19ec1fe122d88a18a127aedb6f5da9aa3"} Sep 10 13:32:00.998834 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:00.998812 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" event={"ID":"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e","Type":"ContainerStarted","Data":"3ca1d31e078a4c1f1e048a8c08349893bf13f8816369e31f1c0bfcf609da7c6e"} Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900611 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:01.900759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.900897 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.900956 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls podName:246361dc-74c9-4643-8740-456a725d7cb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.900938752 +0000 UTC m=+201.969938702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls") pod "dns-default-hmmz9" (UID: "246361dc-74c9-4643-8740-456a725d7cb5") : secret "dns-default-metrics-tls" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901335 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901350 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746bc7545d-7r2l4: secret "image-registry-tls" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901404 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls podName:926a37d0-25e0-4996-8995-de7bd2115b3c nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.901391543 +0000 UTC m=+201.970391490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls") pod "image-registry-746bc7545d-7r2l4" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c") : secret "image-registry-tls" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901460 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901489 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.901480032 +0000 UTC m=+201.970479981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : secret "router-metrics-certs-default" not found Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901548 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.901538023 +0000 UTC m=+201.970537973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : configmap references non-existent config key: service-ca.crt Sep 10 13:32:01.902000 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901614 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:32:01.902971 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901664 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert podName:a8dab21e-4671-4586-9934-9f51c1b6dcfe nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.901646524 +0000 UTC m=+201.970646475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-flw8r" (UID: "a8dab21e-4671-4586-9934-9f51c1b6dcfe") : secret "networking-console-plugin-cert" not found Sep 10 13:32:01.902971 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901719 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:32:01.902971 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901747 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls podName:62e1b7ca-1e07-4f56-96d6-a865772d029f nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.901738436 +0000 UTC m=+201.970738380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nfl8v" (UID: "62e1b7ca-1e07-4f56-96d6-a865772d029f") : secret "samples-operator-tls" not found Sep 10 13:32:01.902971 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901813 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:01.902971 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:01.901853 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls podName:91c4e41b-40d7-4042-8d81-b62e0d488e98 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:03.901833858 +0000 UTC m=+201.970833802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q9rnx" (UID: "91c4e41b-40d7-4042-8d81-b62e0d488e98") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:03.923017 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.922968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:03.923017 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.923024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.923056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.923109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.923162 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.923206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:03.923253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923382 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:32:03.923552 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923443 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.923423427 +0000 UTC m=+205.992423376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : secret "router-metrics-certs-default" not found Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923691 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923760 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls podName:91c4e41b-40d7-4042-8d81-b62e0d488e98 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.923741539 +0000 UTC m=+205.992741496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q9rnx" (UID: "91c4e41b-40d7-4042-8d81-b62e0d488e98") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923837 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.923825225 +0000 UTC m=+205.992825182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : configmap references non-existent config key: service-ca.crt Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923854 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923896 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923898 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls podName:246361dc-74c9-4643-8740-456a725d7cb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.923883826 +0000 UTC m=+205.992883773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls") pod "dns-default-hmmz9" (UID: "246361dc-74c9-4643-8740-456a725d7cb5") : secret "dns-default-metrics-tls" not found Sep 10 13:32:03.923956 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923940 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert podName:a8dab21e-4671-4586-9934-9f51c1b6dcfe nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.923929277 +0000 UTC m=+205.992929225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-flw8r" (UID: "a8dab21e-4671-4586-9934-9f51c1b6dcfe") : secret "networking-console-plugin-cert" not found Sep 10 13:32:03.924325 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923961 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:32:03.924325 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.923973 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746bc7545d-7r2l4: secret "image-registry-tls" not found Sep 10 13:32:03.924325 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.924009 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls podName:926a37d0-25e0-4996-8995-de7bd2115b3c nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.923996705 +0000 UTC m=+205.992996653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls") pod "image-registry-746bc7545d-7r2l4" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c") : secret "image-registry-tls" not found Sep 10 13:32:03.924325 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.924075 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:32:03.924325 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:03.924129 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls podName:62e1b7ca-1e07-4f56-96d6-a865772d029f nodeName:}" failed. No retries permitted until 2026-09-10 13:32:07.924106233 +0000 UTC m=+205.993106195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nfl8v" (UID: "62e1b7ca-1e07-4f56-96d6-a865772d029f") : secret "samples-operator-tls" not found Sep 10 13:32:04.015963 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:04.015465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" event={"ID":"f23db181-9586-4885-ab98-3195ec654d20","Type":"ContainerStarted","Data":"f9b2a91b9f2dfb5be522620baf7ecf0fd1b542d3cd57e0801b23c5cf9b569d90"} Sep 10 13:32:07.960965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.960915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.961098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961111 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.961153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961192 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls podName:91c4e41b-40d7-4042-8d81-b62e0d488e98 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.96116987 +0000 UTC m=+214.030169819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q9rnx" (UID: "91c4e41b-40d7-4042-8d81-b62e0d488e98") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961241 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961250 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.961247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961254 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746bc7545d-7r2l4: secret "image-registry-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961303 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls podName:246361dc-74c9-4643-8740-456a725d7cb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.961287148 +0000 UTC m=+214.030287110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls") pod "dns-default-hmmz9" (UID: "246361dc-74c9-4643-8740-456a725d7cb5") : secret "dns-default-metrics-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961304 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961339 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.961327308 +0000 UTC m=+214.030327257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : secret "router-metrics-certs-default" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961355 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls podName:926a37d0-25e0-4996-8995-de7bd2115b3c nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.96134576 +0000 UTC m=+214.030345703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls") pod "image-registry-746bc7545d-7r2l4" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c") : secret "image-registry-tls" not found Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.961371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.961395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:07.961422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:07.961417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:07.962106 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961499 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:32:07.962106 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961522 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls podName:62e1b7ca-1e07-4f56-96d6-a865772d029f nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.96151513 +0000 UTC m=+214.030515072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nfl8v" (UID: "62e1b7ca-1e07-4f56-96d6-a865772d029f") : secret "samples-operator-tls" not found Sep 10 13:32:07.962106 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961577 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.961569878 +0000 UTC m=+214.030569821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : configmap references non-existent config key: service-ca.crt Sep 10 13:32:07.962106 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961634 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:32:07.962106 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:07.961655 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert podName:a8dab21e-4671-4586-9934-9f51c1b6dcfe nodeName:}" failed. No retries permitted until 2026-09-10 13:32:15.961649616 +0000 UTC m=+214.030649559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-flw8r" (UID: "a8dab21e-4671-4586-9934-9f51c1b6dcfe") : secret "networking-console-plugin-cert" not found Sep 10 13:32:09.031499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.031452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ldfgt" event={"ID":"21675ea4-e8e5-4530-8b0a-397ae6ba0f92","Type":"ContainerStarted","Data":"949a980f3f9a4f5ed6e49ac6ba275df109142c3aa94cb954c3127b5a7b0b928e"} Sep 10 13:32:09.034154 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.034123 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhj7q" event={"ID":"90f4e27e-5d46-4dfe-bec6-38972d80368b","Type":"ContainerStarted","Data":"16211bdf9c8d83ae45a5df3e6bb47098720313f24231c385dac83745b2a348ff"} Sep 10 13:32:09.037532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.037115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" event={"ID":"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e","Type":"ContainerStarted","Data":"ca1edaa3ed5e7144e29e5093d6781c5ad8e7a77e985d3716969cd048e39fc703"} Sep 10 13:32:09.042186 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.039909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" event={"ID":"395969e1-8df4-46fc-bff6-86313e25ab87","Type":"ContainerStarted","Data":"8c0fc26cb9673377f90781e832aa3f838463e61c5ed427de66ff7cd4180bc45e"} Sep 10 13:32:09.043808 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.043784 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/0.log" Sep 10 13:32:09.043903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.043822 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" containerID="66b95294895353c03464f2a8b2615cb67dfcf7f8444b316a58d956ae1c610ec8" exitCode=255 Sep 10 13:32:09.043903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.043887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-56g9k" event={"ID":"ab31e74a-1aef-4a39-a7ee-757e5c05e212","Type":"ContainerDied","Data":"66b95294895353c03464f2a8b2615cb67dfcf7f8444b316a58d956ae1c610ec8"} Sep 10 13:32:09.044169 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.044149 2568 scope.go:117] "RemoveContainer" containerID="66b95294895353c03464f2a8b2615cb67dfcf7f8444b316a58d956ae1c610ec8" Sep 10 13:32:09.048642 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.048563 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ldfgt" podStartSLOduration=64.656766399 podStartE2EDuration="1m13.048546621s" totalImagesPullingTime="8.391780222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:09.047908417 +0000 UTC m=+207.116908383" watchObservedRunningTime="2026-09-10 13:32:09.048546621 +0000 UTC m=+207.117546583" Sep 10 13:32:09.049244 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.049199 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-67l62" podStartSLOduration=66.995582837 podStartE2EDuration="1m9.049185464s" totalImagesPullingTime="2.053602627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:04.047826148 +0000 UTC m=+202.116826114" watchObservedRunningTime="2026-09-10 13:32:09.049185464 +0000 UTC m=+207.118185429" Sep 10 13:32:09.050722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.050690 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" event={"ID":"b0865057-5202-4207-a761-5ec236bf4b84","Type":"ContainerStarted","Data":"0f0691d3465756c00e656b6ec5b8bb12f71d6ca35f612645d77389f514d2a26e"} Sep 10 13:32:09.053779 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.053741 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" event={"ID":"836f29b4-3ff1-47fc-ab08-28dcdff5144c","Type":"ContainerStarted","Data":"a96646e1493f7e305254080423435bb1ccf9625112c087386c8d6d7929c7b34d"} Sep 10 13:32:09.070614 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.069586 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mhj7q" podStartSLOduration=1.283605591 podStartE2EDuration="9.069567374s" totalImagesPullingTime="7.785961783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:09.068428179 +0000 UTC m=+207.137428145" watchObservedRunningTime="2026-09-10 13:32:09.069567374 +0000 UTC m=+207.138567340" Sep 10 13:32:09.098590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.098533 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" podStartSLOduration=185.270470541 podStartE2EDuration="3m13.098512249s" totalImagesPullingTime="7.828041708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:09.096380133 +0000 UTC m=+207.165380103" watchObservedRunningTime="2026-09-10 13:32:09.098512249 +0000 UTC m=+207.167512216" Sep 10 13:32:09.115165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.114726 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-2kfkp" podStartSLOduration=185.288681081 podStartE2EDuration="3m13.114707592s" totalImagesPullingTime="7.826026511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:09.113847828 +0000 UTC m=+207.182847795" watchObservedRunningTime="2026-09-10 13:32:09.114707592 +0000 UTC m=+207.183707562" Sep 10 13:32:09.163647 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.163390 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" podStartSLOduration=185.288242595 podStartE2EDuration="3m13.163357826s" totalImagesPullingTime="7.875115231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:09.162350556 +0000 UTC m=+207.231350523" watchObservedRunningTime="2026-09-10 13:32:09.163357826 +0000 UTC m=+207.232357789" Sep 10 13:32:09.190977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.189928 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" podStartSLOduration=185.345773923 podStartE2EDuration="3m13.189908097s" totalImagesPullingTime="7.844134174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:09.188690404 +0000 UTC m=+207.257690366" watchObservedRunningTime="2026-09-10 13:32:09.189908097 +0000 UTC m=+207.258908063" Sep 10 13:32:09.799785 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.799752 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j5qff"] Sep 10 13:32:09.803415 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.803390 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.805921 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.805897 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 13:32:09.806040 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.805996 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 13:32:09.806188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.806173 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9gkj\"" Sep 10 13:32:09.829675 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.829648 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j5qff"] Sep 10 13:32:09.884716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.884681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.884874 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.884751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.884874 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.884806 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnbv\" (UniqueName: \"kubernetes.io/projected/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-api-access-ljnbv\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.884874 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.884833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04eae699-fb7a-4431-8f00-7997b5bb047a-data-volume\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.885006 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.884919 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04eae699-fb7a-4431-8f00-7997b5bb047a-crio-socket\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04eae699-fb7a-4431-8f00-7997b5bb047a-crio-socket\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985466 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04eae699-fb7a-4431-8f00-7997b5bb047a-crio-socket\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985483 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnbv\" (UniqueName: \"kubernetes.io/projected/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-api-access-ljnbv\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985521 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:09.985511 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:32:09.985801 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04eae699-fb7a-4431-8f00-7997b5bb047a-data-volume\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:09.985801 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:09.985584 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:10.485569193 +0000 UTC m=+208.554569136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : configmap "kube-rbac-proxy" not found Sep 10 13:32:09.985801 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:09.985646 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:32:09.985801 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:09.985742 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:10.485723681 +0000 UTC m=+208.554723629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : secret "insights-runtime-extractor-tls" not found Sep 10 13:32:09.986014 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:09.985866 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04eae699-fb7a-4431-8f00-7997b5bb047a-data-volume\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:10.000926 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.000851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnbv\" (UniqueName: \"kubernetes.io/projected/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-api-access-ljnbv\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:10.059141 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.059115 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/1.log" Sep 10 13:32:10.059658 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.059641 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/0.log" Sep 10 13:32:10.059752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.059681 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" containerID="37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74" exitCode=255 Sep 10 13:32:10.059814 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.059787 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-56g9k" event={"ID":"ab31e74a-1aef-4a39-a7ee-757e5c05e212","Type":"ContainerDied","Data":"37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74"} Sep 10 13:32:10.059862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.059850 2568 scope.go:117] "RemoveContainer" containerID="66b95294895353c03464f2a8b2615cb67dfcf7f8444b316a58d956ae1c610ec8" Sep 10 13:32:10.060083 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.060061 2568 scope.go:117] "RemoveContainer" containerID="37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74" Sep 10 13:32:10.060278 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:10.060257 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-56g9k_openshift-console-operator(ab31e74a-1aef-4a39-a7ee-757e5c05e212)\"" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podUID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" Sep 10 13:32:10.491797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.491755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:10.492002 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.491871 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:10.492110 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:10.492090 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:32:10.492167 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:10.492156 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:11.492135764 +0000 UTC m=+209.561135713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : secret "insights-runtime-extractor-tls" not found Sep 10 13:32:10.492301 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:10.492280 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:32:10.492365 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:10.492354 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:11.492339306 +0000 UTC m=+209.561339256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : configmap "kube-rbac-proxy" not found Sep 10 13:32:10.530160 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.530124 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:10.530315 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.530180 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:10.961679 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.961583 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb"] Sep 10 13:32:10.965985 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.965962 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" Sep 10 13:32:10.968612 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.968577 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 13:32:10.968691 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.968582 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 13:32:10.969533 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.969517 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-769qt\"" Sep 10 13:32:10.985775 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:10.985652 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb"] Sep 10 13:32:11.065063 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.065034 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/1.log" Sep 10 13:32:11.065439 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.065348 2568 scope.go:117] "RemoveContainer" containerID="37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74" Sep 10 13:32:11.065506 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:11.065489 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-56g9k_openshift-console-operator(ab31e74a-1aef-4a39-a7ee-757e5c05e212)\"" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podUID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" Sep 10 13:32:11.097575 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.097549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphbk\" (UniqueName: \"kubernetes.io/projected/bb9f87b5-a383-4c25-a46d-77842b449b0f-kube-api-access-qphbk\") pod \"migrator-fdbb8fdff-s4gxb\" (UID: \"bb9f87b5-a383-4c25-a46d-77842b449b0f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" Sep 10 13:32:11.198418 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.198367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphbk\" (UniqueName: \"kubernetes.io/projected/bb9f87b5-a383-4c25-a46d-77842b449b0f-kube-api-access-qphbk\") pod \"migrator-fdbb8fdff-s4gxb\" (UID: \"bb9f87b5-a383-4c25-a46d-77842b449b0f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" Sep 10 13:32:11.209085 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.209051 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphbk\" (UniqueName: \"kubernetes.io/projected/bb9f87b5-a383-4c25-a46d-77842b449b0f-kube-api-access-qphbk\") pod \"migrator-fdbb8fdff-s4gxb\" (UID: \"bb9f87b5-a383-4c25-a46d-77842b449b0f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" Sep 10 13:32:11.275929 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.275899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" Sep 10 13:32:11.501489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.501447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:11.501680 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.501583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:11.501680 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:11.501613 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:32:11.501680 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:11.501677 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:32:11.501779 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:11.501680 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:13.50166499 +0000 UTC m=+211.570664937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : secret "insights-runtime-extractor-tls" not found Sep 10 13:32:11.501779 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:11.501734 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:13.501716525 +0000 UTC m=+211.570716471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : configmap "kube-rbac-proxy" not found Sep 10 13:32:11.546324 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:11.546250 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb"] Sep 10 13:32:11.549106 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:11.549076 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9f87b5_a383_4c25_a46d_77842b449b0f.slice/crio-ebb9951d9fde334546300fa1925358029e19e2769f7c3c6b8cd5b25dc1146408 WatchSource:0}: Error finding container ebb9951d9fde334546300fa1925358029e19e2769f7c3c6b8cd5b25dc1146408: Status 404 returned error can't find the container with id ebb9951d9fde334546300fa1925358029e19e2769f7c3c6b8cd5b25dc1146408 Sep 10 13:32:12.069846 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.069805 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" event={"ID":"bb9f87b5-a383-4c25-a46d-77842b449b0f","Type":"ContainerStarted","Data":"ebb9951d9fde334546300fa1925358029e19e2769f7c3c6b8cd5b25dc1146408"} Sep 10 13:32:12.070312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.070138 2568 scope.go:117] "RemoveContainer" containerID="37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74" Sep 10 13:32:12.070352 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:12.070318 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-56g9k_openshift-console-operator(ab31e74a-1aef-4a39-a7ee-757e5c05e212)\"" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podUID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" Sep 10 13:32:12.669706 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.669682 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-42tjr"] Sep 10 13:32:12.673741 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.673720 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.676589 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.676571 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 13:32:12.676698 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.676651 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nndxl\"" Sep 10 13:32:12.677625 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.677581 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 13:32:12.677719 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.677682 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 13:32:12.677775 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.677736 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 13:32:12.690896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.690872 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-42tjr"] Sep 10 13:32:12.813674 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.813639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5bk\" (UniqueName: \"kubernetes.io/projected/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-kube-api-access-8w5bk\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.813870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.813831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-signing-key\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.813870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.813863 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-signing-cabundle\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.915170 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.915081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-signing-key\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.915170 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.915125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-signing-cabundle\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.915364 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.915278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5bk\" (UniqueName: \"kubernetes.io/projected/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-kube-api-access-8w5bk\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.915756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.915737 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-signing-cabundle\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.917519 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.917499 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-signing-key\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.929096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.929073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5bk\" (UniqueName: \"kubernetes.io/projected/2e4eda87-8a8a-4e2f-82c6-a7f784f213e4-kube-api-access-8w5bk\") pod \"service-ca-75d99c484-42tjr\" (UID: \"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4\") " pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:12.985226 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:12.985191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-42tjr" Sep 10 13:32:13.075811 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:13.075780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" event={"ID":"bb9f87b5-a383-4c25-a46d-77842b449b0f","Type":"ContainerStarted","Data":"b02d46fca91a51e28e240bcfb4cc3802341118daeda674ac511e5dd88893bbf9"} Sep 10 13:32:13.076171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:13.075832 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" event={"ID":"bb9f87b5-a383-4c25-a46d-77842b449b0f","Type":"ContainerStarted","Data":"ccb5c2fc5d085ab1c179c9fa9e2eb4ee8fa3308ad79026ac1563366a8a890948"} Sep 10 13:32:13.106147 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:13.106098 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s4gxb" podStartSLOduration=2.024998101 podStartE2EDuration="3.10608272s" totalImagesPullingTime="1.081084619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:13.106005669 +0000 UTC m=+211.175005628" watchObservedRunningTime="2026-09-10 13:32:13.10608272 +0000 UTC m=+211.175082684" Sep 10 13:32:13.262136 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:13.260782 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-42tjr"] Sep 10 13:32:13.262136 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:13.261350 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4eda87_8a8a_4e2f_82c6_a7f784f213e4.slice/crio-7078df157aaf418969f194994ac710b95e3483817553c111da97ded4e0099f5f WatchSource:0}: Error finding container 7078df157aaf418969f194994ac710b95e3483817553c111da97ded4e0099f5f: Status 404 returned error can't find the container with id 7078df157aaf418969f194994ac710b95e3483817553c111da97ded4e0099f5f Sep 10 13:32:13.520558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:13.520457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:13.520558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:13.520541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:13.520834 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:13.520634 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:32:13.520834 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:13.520714 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:32:13.520834 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:13.520717 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:17.520696127 +0000 UTC m=+215.589696089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : configmap "kube-rbac-proxy" not found Sep 10 13:32:13.520834 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:13.520788 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:17.520772458 +0000 UTC m=+215.589772408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : secret "insights-runtime-extractor-tls" not found Sep 10 13:32:14.080576 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:14.080538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-42tjr" event={"ID":"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4","Type":"ContainerStarted","Data":"b500284cd11a94b879c2eb286532b18a9276686a5af9deb5b01d61b170544000"} Sep 10 13:32:14.081075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:14.080593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-42tjr" event={"ID":"2e4eda87-8a8a-4e2f-82c6-a7f784f213e4","Type":"ContainerStarted","Data":"7078df157aaf418969f194994ac710b95e3483817553c111da97ded4e0099f5f"} Sep 10 13:32:14.095238 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:14.095196 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-42tjr" podStartSLOduration=2.095181543 podStartE2EDuration="2.095181543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:14.094819585 +0000 UTC m=+212.163819549" watchObservedRunningTime="2026-09-10 13:32:14.095181543 +0000 UTC m=+212.164181507" Sep 10 13:32:14.110287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:14.110264 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlrlg_7f6499f9-7d26-43d2-a1b9-f86c25c9ce80/dns-node-resolver/0.log" Sep 10 13:32:14.918150 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:14.918120 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cw5l8_c8a458ea-1aef-4cd9-b83e-70477e42d196/node-ca/0.log" Sep 10 13:32:15.714424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:15.714374 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhj7q_90f4e27e-5d46-4dfe-bec6-38972d80368b/serve-healthcheck-canary/0.log" Sep 10 13:32:16.046365 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:16.046545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046377 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:16.046545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:16.046545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:16.046545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:16.046545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046518 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:16.046545 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046487 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046566 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.046571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046573 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046503 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046646 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:32.046612537 +0000 UTC m=+230.115612494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : secret "router-metrics-certs-default" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046646 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746bc7545d-7r2l4: secret "image-registry-tls" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046692 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls podName:926a37d0-25e0-4996-8995-de7bd2115b3c nodeName:}" failed. No retries permitted until 2026-09-10 13:32:32.046678038 +0000 UTC m=+230.115677992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls") pod "image-registry-746bc7545d-7r2l4" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c") : secret "image-registry-tls" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046646 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046709 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls podName:246361dc-74c9-4643-8740-456a725d7cb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:32.046698734 +0000 UTC m=+230.115698681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls") pod "dns-default-hmmz9" (UID: "246361dc-74c9-4643-8740-456a725d7cb5") : secret "dns-default-metrics-tls" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046726 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle podName:9d949756-5384-47bf-a2e0-4380b00b5e5a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:32.046717563 +0000 UTC m=+230.115717507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle") pod "router-default-64447d557d-vpzh2" (UID: "9d949756-5384-47bf-a2e0-4380b00b5e5a") : configmap references non-existent config key: service-ca.crt Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046743 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert podName:a8dab21e-4671-4586-9934-9f51c1b6dcfe nodeName:}" failed. No retries permitted until 2026-09-10 13:32:32.046735043 +0000 UTC m=+230.115734990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-flw8r" (UID: "a8dab21e-4671-4586-9934-9f51c1b6dcfe") : secret "networking-console-plugin-cert" not found Sep 10 13:32:16.046790 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:16.046763 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls podName:91c4e41b-40d7-4042-8d81-b62e0d488e98 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:32.046754274 +0000 UTC m=+230.115754225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q9rnx" (UID: "91c4e41b-40d7-4042-8d81-b62e0d488e98") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:32:16.048939 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.048920 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62e1b7ca-1e07-4f56-96d6-a865772d029f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nfl8v\" (UID: \"62e1b7ca-1e07-4f56-96d6-a865772d029f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:16.095249 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.095216 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" Sep 10 13:32:16.365974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:16.365946 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v"] Sep 10 13:32:17.096130 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.096089 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" event={"ID":"62e1b7ca-1e07-4f56-96d6-a865772d029f","Type":"ContainerStarted","Data":"9be5f9b1e82c9bf0e1ffcd288689c75547a0b0da37f54cbd9c79093b22ee08f3"} Sep 10 13:32:17.310889 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.310864 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/egress-router-binary-copy/0.log" Sep 10 13:32:17.511372 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.511330 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/cni-plugins/0.log" Sep 10 13:32:17.562489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.562451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:17.562702 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.562525 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:17.562702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:17.562589 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:32:17.562702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:17.562685 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:25.562664015 +0000 UTC m=+223.631663975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : configmap "kube-rbac-proxy" not found Sep 10 13:32:17.562702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:17.562689 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:32:17.562906 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:17.562745 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:25.562729233 +0000 UTC m=+223.631729188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : secret "insights-runtime-extractor-tls" not found Sep 10 13:32:17.710862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.710832 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/bond-cni-plugin/0.log" Sep 10 13:32:17.911180 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:17.911103 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/routeoverride-cni/0.log" Sep 10 13:32:18.113349 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:18.113318 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/whereabouts-cni-bincopy/0.log" Sep 10 13:32:18.312230 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:18.312214 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/whereabouts-cni/0.log" Sep 10 13:32:18.508908 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:18.508879 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwdlg_cabbefbe-25a5-4ab5-b815-8e0b2b6856f3/kube-multus-additional-cni-plugins/0.log" Sep 10 13:32:19.105574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:19.105539 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" event={"ID":"62e1b7ca-1e07-4f56-96d6-a865772d029f","Type":"ContainerStarted","Data":"5fa3674314d0e4a67995de34100ddc973a0f2624029147b5c2af13832394f0f4"} Sep 10 13:32:19.105776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:19.105653 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" event={"ID":"62e1b7ca-1e07-4f56-96d6-a865772d029f","Type":"ContainerStarted","Data":"42ed63a17bcb6b7f78777fc3383ffa6dbfb3e70be8b3849254d325f6db0e4289"} Sep 10 13:32:19.123400 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:19.123350 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nfl8v" podStartSLOduration=201.232183748 podStartE2EDuration="3m23.123336294s" totalImagesPullingTime="1.891152546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:19.123050273 +0000 UTC m=+217.192050250" watchObservedRunningTime="2026-09-10 13:32:19.123336294 +0000 UTC m=+217.192336259" Sep 10 13:32:20.109171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:20.109143 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:32:20.311615 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:20.311576 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/2.log" Sep 10 13:32:25.638879 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:25.638840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:25.639469 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:25.638918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:25.639469 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:25.639037 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:32:25.639469 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:25.639132 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls podName:04eae699-fb7a-4431-8f00-7997b5bb047a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:41.639111016 +0000 UTC m=+239.708110973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j5qff" (UID: "04eae699-fb7a-4431-8f00-7997b5bb047a") : secret "insights-runtime-extractor-tls" not found Sep 10 13:32:25.639469 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:25.639359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04eae699-fb7a-4431-8f00-7997b5bb047a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:25.714473 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:25.714443 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/ovn-controller/0.log" Sep 10 13:32:25.910745 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:25.910681 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/ovn-acl-logging/0.log" Sep 10 13:32:26.111932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:26.111902 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/kube-rbac-proxy-node/0.log" Sep 10 13:32:26.311426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:26.311395 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:32:26.513295 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:26.513264 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/northd/0.log" Sep 10 13:32:26.711169 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:26.711095 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/nbdb/0.log" Sep 10 13:32:26.910691 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:26.910664 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/sbdb/0.log" Sep 10 13:32:27.116118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:27.116083 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz9xv_a077ae2e-221e-4a68-847f-0dae1c4abf14/ovnkube-controller/0.log" Sep 10 13:32:27.310587 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:27.310562 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-2kfkp_395969e1-8df4-46fc-bff6-86313e25ab87/check-endpoints/0.log" Sep 10 13:32:27.418534 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:27.418453 2568 scope.go:117] "RemoveContainer" containerID="37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74" Sep 10 13:32:28.136252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.136225 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:32:28.136685 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.136617 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/1.log" Sep 10 13:32:28.136685 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.136647 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" containerID="0b45f80042172e3252e466322cc82d300b59bf08ce68d3e765310de1db10a342" exitCode=255 Sep 10 13:32:28.136791 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.136703 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-56g9k" event={"ID":"ab31e74a-1aef-4a39-a7ee-757e5c05e212","Type":"ContainerDied","Data":"0b45f80042172e3252e466322cc82d300b59bf08ce68d3e765310de1db10a342"} Sep 10 13:32:28.136791 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.136748 2568 scope.go:117] "RemoveContainer" containerID="37e0982b26bfbc85ad4154a6496b006e7bb518ab700abb3316ccd80841669e74" Sep 10 13:32:28.137112 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.137096 2568 scope.go:117] "RemoveContainer" containerID="0b45f80042172e3252e466322cc82d300b59bf08ce68d3e765310de1db10a342" Sep 10 13:32:28.137293 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:28.137272 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-56g9k_openshift-console-operator(ab31e74a-1aef-4a39-a7ee-757e5c05e212)\"" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podUID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" Sep 10 13:32:28.312252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:28.312224 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8w67r_d96bfd07-ac79-4a08-9194-212590d011d9/iptables-alerter/0.log" Sep 10 13:32:29.141990 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:29.141966 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:32:30.529996 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:30.529957 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:30.530392 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:30.530024 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:30.530392 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:30.530320 2568 scope.go:117] "RemoveContainer" containerID="0b45f80042172e3252e466322cc82d300b59bf08ce68d3e765310de1db10a342" Sep 10 13:32:30.530490 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:30.530471 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-56g9k_openshift-console-operator(ab31e74a-1aef-4a39-a7ee-757e5c05e212)\"" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podUID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" Sep 10 13:32:32.104864 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.104824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:32.104864 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.104864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:32.105431 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.104911 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:32.105431 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.104949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:32.105431 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.104990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:32.105431 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.105165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:32.105677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.105623 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d949756-5384-47bf-a2e0-4380b00b5e5a-service-ca-bundle\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:32.107656 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.107622 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8dab21e-4671-4586-9934-9f51c1b6dcfe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-flw8r\" (UID: \"a8dab21e-4671-4586-9934-9f51c1b6dcfe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:32.107768 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.107698 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c4e41b-40d7-4042-8d81-b62e0d488e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q9rnx\" (UID: \"91c4e41b-40d7-4042-8d81-b62e0d488e98\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:32.107768 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.107755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/246361dc-74c9-4643-8740-456a725d7cb5-metrics-tls\") pod \"dns-default-hmmz9\" (UID: \"246361dc-74c9-4643-8740-456a725d7cb5\") " pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:32.107843 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.107767 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d949756-5384-47bf-a2e0-4380b00b5e5a-metrics-certs\") pod \"router-default-64447d557d-vpzh2\" (UID: \"9d949756-5384-47bf-a2e0-4380b00b5e5a\") " pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:32.107843 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.107816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"image-registry-746bc7545d-7r2l4\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:32.301754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.301715 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:32.319094 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.319064 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" Sep 10 13:32:32.345255 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.345225 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" Sep 10 13:32:32.352404 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.352380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:32.377323 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.377287 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:32.650168 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.650086 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-746bc7545d-7r2l4"] Sep 10 13:32:32.653655 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:32.653494 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926a37d0_25e0_4996_8995_de7bd2115b3c.slice/crio-90603d24cf62a172677916d18e731d30cf59b6f7e9af9ae00a8c907ea95cfd8f WatchSource:0}: Error finding container 90603d24cf62a172677916d18e731d30cf59b6f7e9af9ae00a8c907ea95cfd8f: Status 404 returned error can't find the container with id 90603d24cf62a172677916d18e731d30cf59b6f7e9af9ae00a8c907ea95cfd8f Sep 10 13:32:32.654126 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.654043 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx"] Sep 10 13:32:32.655915 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:32.655884 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c4e41b_40d7_4042_8d81_b62e0d488e98.slice/crio-67fcfb129fc5d0a443f7c5709e322a7943c270b9d607fe0a1b1876d8ec16443a WatchSource:0}: Error finding container 67fcfb129fc5d0a443f7c5709e322a7943c270b9d607fe0a1b1876d8ec16443a: Status 404 returned error can't find the container with id 67fcfb129fc5d0a443f7c5709e322a7943c270b9d607fe0a1b1876d8ec16443a Sep 10 13:32:32.682127 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.682093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-flw8r"] Sep 10 13:32:32.684306 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:32.684278 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dab21e_4671_4586_9934_9f51c1b6dcfe.slice/crio-875e8b39a3a686b0377d3d9dfee83cf9b973d6c1521ea93ea550cb08ce6f142b WatchSource:0}: Error finding container 875e8b39a3a686b0377d3d9dfee83cf9b973d6c1521ea93ea550cb08ce6f142b: Status 404 returned error can't find the container with id 875e8b39a3a686b0377d3d9dfee83cf9b973d6c1521ea93ea550cb08ce6f142b Sep 10 13:32:32.701492 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.701467 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmmz9"] Sep 10 13:32:32.710670 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:32.710649 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246361dc_74c9_4643_8740_456a725d7cb5.slice/crio-87a3c72bf93c583dad851718ed6107007232fad42b83a6f4183a8ec4e73f7841 WatchSource:0}: Error finding container 87a3c72bf93c583dad851718ed6107007232fad42b83a6f4183a8ec4e73f7841: Status 404 returned error can't find the container with id 87a3c72bf93c583dad851718ed6107007232fad42b83a6f4183a8ec4e73f7841 Sep 10 13:32:32.736863 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:32.736799 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64447d557d-vpzh2"] Sep 10 13:32:32.739437 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:32.739413 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d949756_5384_47bf_a2e0_4380b00b5e5a.slice/crio-7b171c91bdcf83a1b192e94c7b90a5ce322647a20c926595e23a406d78476173 WatchSource:0}: Error finding container 7b171c91bdcf83a1b192e94c7b90a5ce322647a20c926595e23a406d78476173: Status 404 returned error can't find the container with id 7b171c91bdcf83a1b192e94c7b90a5ce322647a20c926595e23a406d78476173 Sep 10 13:32:33.163491 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.163458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmmz9" event={"ID":"246361dc-74c9-4643-8740-456a725d7cb5","Type":"ContainerStarted","Data":"87a3c72bf93c583dad851718ed6107007232fad42b83a6f4183a8ec4e73f7841"} Sep 10 13:32:33.164787 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.164756 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" event={"ID":"91c4e41b-40d7-4042-8d81-b62e0d488e98","Type":"ContainerStarted","Data":"67fcfb129fc5d0a443f7c5709e322a7943c270b9d607fe0a1b1876d8ec16443a"} Sep 10 13:32:33.166326 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.166295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64447d557d-vpzh2" event={"ID":"9d949756-5384-47bf-a2e0-4380b00b5e5a","Type":"ContainerStarted","Data":"b4582339d1658d864a2cab8d65ef6e49b339120067edbad877cae9ce9c90e996"} Sep 10 13:32:33.166441 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.166356 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64447d557d-vpzh2" event={"ID":"9d949756-5384-47bf-a2e0-4380b00b5e5a","Type":"ContainerStarted","Data":"7b171c91bdcf83a1b192e94c7b90a5ce322647a20c926595e23a406d78476173"} Sep 10 13:32:33.167408 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.167388 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" event={"ID":"a8dab21e-4671-4586-9934-9f51c1b6dcfe","Type":"ContainerStarted","Data":"875e8b39a3a686b0377d3d9dfee83cf9b973d6c1521ea93ea550cb08ce6f142b"} Sep 10 13:32:33.168879 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.168859 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" event={"ID":"926a37d0-25e0-4996-8995-de7bd2115b3c","Type":"ContainerStarted","Data":"b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e"} Sep 10 13:32:33.168991 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.168889 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" event={"ID":"926a37d0-25e0-4996-8995-de7bd2115b3c","Type":"ContainerStarted","Data":"90603d24cf62a172677916d18e731d30cf59b6f7e9af9ae00a8c907ea95cfd8f"} Sep 10 13:32:33.169061 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.168996 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:33.193077 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.193034 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64447d557d-vpzh2" podStartSLOduration=217.193014896 podStartE2EDuration="3m37.193014896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:33.192401899 +0000 UTC m=+231.261401865" watchObservedRunningTime="2026-09-10 13:32:33.193014896 +0000 UTC m=+231.262014860" Sep 10 13:32:33.215543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.215485 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" podStartSLOduration=218.215463131 podStartE2EDuration="3m38.215463131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:33.215053864 +0000 UTC m=+231.284053830" watchObservedRunningTime="2026-09-10 13:32:33.215463131 +0000 UTC m=+231.284463097" Sep 10 13:32:33.353006 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.352966 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:33.355909 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:33.355881 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:34.173930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:34.173867 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:34.175328 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:34.175301 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64447d557d-vpzh2" Sep 10 13:32:36.184481 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:36.184447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmmz9" event={"ID":"246361dc-74c9-4643-8740-456a725d7cb5","Type":"ContainerStarted","Data":"956ee0b09459f0b6bce70afb9f3ded625d966c6c16559567c7ad7bcafc426da2"} Sep 10 13:32:36.184959 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:36.184511 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmmz9" event={"ID":"246361dc-74c9-4643-8740-456a725d7cb5","Type":"ContainerStarted","Data":"860c436810e7bbb27dc2eba962dbe1c96a112c00279d4129b6afca9c8ac46069"} Sep 10 13:32:36.184959 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:36.184543 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:36.185975 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:36.185945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" event={"ID":"91c4e41b-40d7-4042-8d81-b62e0d488e98","Type":"ContainerStarted","Data":"c46d0e4e9a5af5ccccb548d6f9f484f80a6d980799b07412e37ce6a50c55187c"} Sep 10 13:32:36.209580 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:36.209534 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hmmz9" podStartSLOduration=33.578541173 podStartE2EDuration="36.209519236s" totalImagesPullingTime="2.630978063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:36.207980677 +0000 UTC m=+234.276980643" watchObservedRunningTime="2026-09-10 13:32:36.209519236 +0000 UTC m=+234.278519201" Sep 10 13:32:37.190573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:37.190532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" event={"ID":"a8dab21e-4671-4586-9934-9f51c1b6dcfe","Type":"ContainerStarted","Data":"89bfe112e29a776502978f03d0d40ec7719551c92456a21fe2fe8844a123db31"} Sep 10 13:32:37.237592 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:37.237533 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-flw8r" podStartSLOduration=217.58506037 podStartE2EDuration="3m41.237512152s" totalImagesPullingTime="3.652451782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:37.236040852 +0000 UTC m=+235.305040817" watchObservedRunningTime="2026-09-10 13:32:37.237512152 +0000 UTC m=+235.306512117" Sep 10 13:32:37.238281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:37.238251 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q9rnx" podStartSLOduration=218.547316852 podStartE2EDuration="3m41.238242352s" totalImagesPullingTime="2.6909255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:36.226877598 +0000 UTC m=+234.295877564" watchObservedRunningTime="2026-09-10 13:32:37.238242352 +0000 UTC m=+235.307242316" Sep 10 13:32:41.417966 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:41.417931 2568 scope.go:117] "RemoveContainer" containerID="0b45f80042172e3252e466322cc82d300b59bf08ce68d3e765310de1db10a342" Sep 10 13:32:41.418432 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:32:41.418133 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-56g9k_openshift-console-operator(ab31e74a-1aef-4a39-a7ee-757e5c05e212)\"" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podUID="ab31e74a-1aef-4a39-a7ee-757e5c05e212" Sep 10 13:32:41.692866 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:41.692764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:41.695077 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:41.695043 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04eae699-fb7a-4431-8f00-7997b5bb047a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j5qff\" (UID: \"04eae699-fb7a-4431-8f00-7997b5bb047a\") " pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:41.913087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:41.913049 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j5qff" Sep 10 13:32:42.199552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:42.199516 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j5qff"] Sep 10 13:32:42.202588 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:42.202562 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04eae699_fb7a_4431_8f00_7997b5bb047a.slice/crio-eb6a968d0535b015afde987da695ff820a20af61986d93fee87d09662b1d6ee4 WatchSource:0}: Error finding container eb6a968d0535b015afde987da695ff820a20af61986d93fee87d09662b1d6ee4: Status 404 returned error can't find the container with id eb6a968d0535b015afde987da695ff820a20af61986d93fee87d09662b1d6ee4 Sep 10 13:32:42.208684 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:42.208658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j5qff" event={"ID":"04eae699-fb7a-4431-8f00-7997b5bb047a","Type":"ContainerStarted","Data":"eb6a968d0535b015afde987da695ff820a20af61986d93fee87d09662b1d6ee4"} Sep 10 13:32:43.213777 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:43.213689 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j5qff" event={"ID":"04eae699-fb7a-4431-8f00-7997b5bb047a","Type":"ContainerStarted","Data":"cb00eabf1081cb7169d10f1489bceca68a61a961809692539903eea39c5a6e29"} Sep 10 13:32:43.213777 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:43.213744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j5qff" event={"ID":"04eae699-fb7a-4431-8f00-7997b5bb047a","Type":"ContainerStarted","Data":"1de4eed7362faea72ecfb3ea7b9af384e5845eab1a055e945025685e806bcbed"} Sep 10 13:32:45.223939 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:45.223909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j5qff" event={"ID":"04eae699-fb7a-4431-8f00-7997b5bb047a","Type":"ContainerStarted","Data":"45f483a26dd369f900158be10a22180924b3f5b6eebad219ba7b3dd84c503ba8"} Sep 10 13:32:45.251178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:45.251120 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j5qff" podStartSLOduration=34.372588397 podStartE2EDuration="36.251103849s" totalImagesPullingTime="1.878515452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:32:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:45.249593381 +0000 UTC m=+243.318593370" watchObservedRunningTime="2026-09-10 13:32:45.251103849 +0000 UTC m=+243.320103813" Sep 10 13:32:46.193165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.193138 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmmz9" Sep 10 13:32:46.720445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.717883 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-trth9"] Sep 10 13:32:46.725340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.725309 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.731519 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.731492 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 13:32:46.731981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.731672 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 13:32:46.731981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.731725 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 13:32:46.732671 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.732652 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6fhr\"" Sep 10 13:32:46.732936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.732712 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 13:32:46.837675 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-sys\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837675 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-textfile\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50dbf940-291f-49cf-aacd-90a1b04b3a6a-metrics-client-ca\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837786 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-root\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scn47\" (UniqueName: \"kubernetes.io/projected/50dbf940-291f-49cf-aacd-90a1b04b3a6a-kube-api-access-scn47\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.837880 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-tls\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.838066 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.837910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-wtmp\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938631 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50dbf940-291f-49cf-aacd-90a1b04b3a6a-metrics-client-ca\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-root\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scn47\" (UniqueName: \"kubernetes.io/projected/50dbf940-291f-49cf-aacd-90a1b04b3a6a-kube-api-access-scn47\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-tls\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-wtmp\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.938838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938784 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-root\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-sys\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-textfile\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.938966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-sys\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939171 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.939073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-wtmp\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939412 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.939266 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-textfile\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939471 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.939440 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50dbf940-291f-49cf-aacd-90a1b04b3a6a-metrics-client-ca\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.939526 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.939479 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.941187 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.941158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.941292 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.941239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50dbf940-291f-49cf-aacd-90a1b04b3a6a-node-exporter-tls\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:46.949106 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:46.949083 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scn47\" (UniqueName: \"kubernetes.io/projected/50dbf940-291f-49cf-aacd-90a1b04b3a6a-kube-api-access-scn47\") pod \"node-exporter-trth9\" (UID: \"50dbf940-291f-49cf-aacd-90a1b04b3a6a\") " pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:47.036788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:47.036703 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-trth9" Sep 10 13:32:47.048533 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:32:47.048508 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dbf940_291f_49cf_aacd_90a1b04b3a6a.slice/crio-03aeb9b66c4ac6f81c5e4bb4106031b3a3139b998ddf9294761777bda6ab3bfc WatchSource:0}: Error finding container 03aeb9b66c4ac6f81c5e4bb4106031b3a3139b998ddf9294761777bda6ab3bfc: Status 404 returned error can't find the container with id 03aeb9b66c4ac6f81c5e4bb4106031b3a3139b998ddf9294761777bda6ab3bfc Sep 10 13:32:47.232631 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:47.232579 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trth9" event={"ID":"50dbf940-291f-49cf-aacd-90a1b04b3a6a","Type":"ContainerStarted","Data":"03aeb9b66c4ac6f81c5e4bb4106031b3a3139b998ddf9294761777bda6ab3bfc"} Sep 10 13:32:48.238167 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:48.238134 2568 generic.go:356] "Generic (PLEG): container finished" podID="50dbf940-291f-49cf-aacd-90a1b04b3a6a" containerID="9a6ce2f9c9c9cf2cad94a76ff03996ae0975bb799d31dd5a78055aad9018cca7" exitCode=0 Sep 10 13:32:48.238627 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:48.238187 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trth9" event={"ID":"50dbf940-291f-49cf-aacd-90a1b04b3a6a","Type":"ContainerDied","Data":"9a6ce2f9c9c9cf2cad94a76ff03996ae0975bb799d31dd5a78055aad9018cca7"} Sep 10 13:32:49.244095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:49.244055 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trth9" event={"ID":"50dbf940-291f-49cf-aacd-90a1b04b3a6a","Type":"ContainerStarted","Data":"512c6f413d45d06281cd91095bcc63b2f4f7241efb5f44550a0bf8162b9752b3"} Sep 10 13:32:49.244478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:49.244112 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trth9" event={"ID":"50dbf940-291f-49cf-aacd-90a1b04b3a6a","Type":"ContainerStarted","Data":"f79ac7bb6e936f7154d776fafd51e28f1ef2f80efa3fec2332554535d46cd657"} Sep 10 13:32:49.301941 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:49.301887 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-trth9" podStartSLOduration=2.242673937 podStartE2EDuration="3.301870253s" totalImagesPullingTime="998.066988ms" totalInitContainerRuntime="61.129328ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:49.300182664 +0000 UTC m=+247.369182630" watchObservedRunningTime="2026-09-10 13:32:49.301870253 +0000 UTC m=+247.370870218" Sep 10 13:32:52.306219 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:52.306184 2568 patch_prober.go:28] interesting pod/image-registry-746bc7545d-7r2l4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 13:32:52.306635 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:52.306238 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" podUID="926a37d0-25e0-4996-8995-de7bd2115b3c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:32:54.178630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:54.178587 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:32:54.418181 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:54.418146 2568 scope.go:117] "RemoveContainer" containerID="0b45f80042172e3252e466322cc82d300b59bf08ce68d3e765310de1db10a342" Sep 10 13:32:55.272071 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:55.272044 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:32:55.272461 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:55.272133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-56g9k" event={"ID":"ab31e74a-1aef-4a39-a7ee-757e5c05e212","Type":"ContainerStarted","Data":"7bcfee6fe8b19d36cc9b3f62928686875e6b79e2a1e75f8dbffad5482d4bd7d2"} Sep 10 13:32:55.272461 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:55.272431 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:55.277254 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:55.277233 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-56g9k" Sep 10 13:32:55.299724 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:55.299675 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-56g9k" podStartSLOduration=231.395658144 podStartE2EDuration="3m59.299660806s" totalImagesPullingTime="7.904002662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:55.298492812 +0000 UTC m=+253.367492777" watchObservedRunningTime="2026-09-10 13:32:55.299660806 +0000 UTC m=+253.368660771" Sep 10 13:32:59.858028 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:32:59.857995 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-746bc7545d-7r2l4"] Sep 10 13:33:09.136365 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.136276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:33:09.139454 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.139436 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:33:09.148758 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.148726 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162d0641-14f2-41f9-90fe-0c489578f4c6-metrics-certs\") pod \"network-metrics-daemon-2zrmg\" (UID: \"162d0641-14f2-41f9-90fe-0c489578f4c6\") " pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:33:09.336326 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.336294 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4vqdt\"" Sep 10 13:33:09.338609 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.338575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:33:09.341087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.341068 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ml2w\" (UniqueName: \"kubernetes.io/projected/d2874ad8-f365-42de-a179-f209937bc343-kube-api-access-2ml2w\") pod \"network-check-target-zwfnn\" (UID: \"d2874ad8-f365-42de-a179-f209937bc343\") " pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:33:09.344806 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.344792 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2zrmg" Sep 10 13:33:09.623342 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.623311 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2zrmg"] Sep 10 13:33:09.627082 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:33:09.627058 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162d0641_14f2_41f9_90fe_0c489578f4c6.slice/crio-8ec8430b318fddbac2fb7aa86ca4c75fe5a9c4e02294575e094a824df4dc69d1 WatchSource:0}: Error finding container 8ec8430b318fddbac2fb7aa86ca4c75fe5a9c4e02294575e094a824df4dc69d1: Status 404 returned error can't find the container with id 8ec8430b318fddbac2fb7aa86ca4c75fe5a9c4e02294575e094a824df4dc69d1 Sep 10 13:33:09.630756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.630735 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdbpg\"" Sep 10 13:33:09.638802 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.638778 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:33:09.915581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:09.915558 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zwfnn"] Sep 10 13:33:09.917463 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:33:09.917436 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2874ad8_f365_42de_a179_f209937bc343.slice/crio-96650f97877e55a8750ee65790d97ffbf0a4bc374ed737fa9bc39872f73e20ef WatchSource:0}: Error finding container 96650f97877e55a8750ee65790d97ffbf0a4bc374ed737fa9bc39872f73e20ef: Status 404 returned error can't find the container with id 96650f97877e55a8750ee65790d97ffbf0a4bc374ed737fa9bc39872f73e20ef Sep 10 13:33:10.327893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:10.327854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2zrmg" event={"ID":"162d0641-14f2-41f9-90fe-0c489578f4c6","Type":"ContainerStarted","Data":"8ec8430b318fddbac2fb7aa86ca4c75fe5a9c4e02294575e094a824df4dc69d1"} Sep 10 13:33:10.329308 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:10.329281 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zwfnn" event={"ID":"d2874ad8-f365-42de-a179-f209937bc343","Type":"ContainerStarted","Data":"a02ca25a045ba4f3fc4f1af4a83b4291e8f5fb5dcc412fc65a5ae33fd445dc24"} Sep 10 13:33:10.329437 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:10.329328 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zwfnn" event={"ID":"d2874ad8-f365-42de-a179-f209937bc343","Type":"ContainerStarted","Data":"96650f97877e55a8750ee65790d97ffbf0a4bc374ed737fa9bc39872f73e20ef"} Sep 10 13:33:10.329437 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:10.329363 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:33:10.364509 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:10.364464 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zwfnn" podStartSLOduration=255.364449476 podStartE2EDuration="4m15.364449476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:10.363182761 +0000 UTC m=+268.432182725" watchObservedRunningTime="2026-09-10 13:33:10.364449476 +0000 UTC m=+268.433449440" Sep 10 13:33:11.335358 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:11.335258 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2zrmg" event={"ID":"162d0641-14f2-41f9-90fe-0c489578f4c6","Type":"ContainerStarted","Data":"6b6f3171f0235fd2857684709f6d3d8eba6d38fada44da388448caa994cb9052"} Sep 10 13:33:11.335358 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:11.335326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2zrmg" event={"ID":"162d0641-14f2-41f9-90fe-0c489578f4c6","Type":"ContainerStarted","Data":"383f28a6007f9365c8d4a63019dcc3639af775b2d2f721bb1f75a3cb6bedb75b"} Sep 10 13:33:11.357444 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:11.357392 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2zrmg" podStartSLOduration=254.944598365 podStartE2EDuration="4m16.357377506s" totalImagesPullingTime="1.412779141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:11.355706177 +0000 UTC m=+269.424706142" watchObservedRunningTime="2026-09-10 13:33:11.357377506 +0000 UTC m=+269.426377471" Sep 10 13:33:20.369789 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:20.369759 2568 generic.go:356] "Generic (PLEG): container finished" podID="14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e" containerID="ca1edaa3ed5e7144e29e5093d6781c5ad8e7a77e985d3716969cd048e39fc703" exitCode=0 Sep 10 13:33:20.370163 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:20.369812 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" event={"ID":"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e","Type":"ContainerDied","Data":"ca1edaa3ed5e7144e29e5093d6781c5ad8e7a77e985d3716969cd048e39fc703"} Sep 10 13:33:20.370163 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:20.370095 2568 scope.go:117] "RemoveContainer" containerID="ca1edaa3ed5e7144e29e5093d6781c5ad8e7a77e985d3716969cd048e39fc703" Sep 10 13:33:21.376039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:21.375998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pbltt" event={"ID":"14bfda3c-9cfd-4563-9e9c-a0fccb15cc9e","Type":"ContainerStarted","Data":"eb7ac61b068e78317c19e23e6e23452326318d0deec6eb80b79c2454f7a50260"} Sep 10 13:33:24.881903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:24.881858 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" podUID="926a37d0-25e0-4996-8995-de7bd2115b3c" containerName="registry" containerID="cri-o://b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e" gracePeriod=30 Sep 10 13:33:25.123152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.123128 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:33:25.157723 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157652 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/926a37d0-25e0-4996-8995-de7bd2115b3c-ca-trust-extracted\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157723 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157684 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-certificates\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157723 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157717 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-installation-pull-secrets\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157733 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-trusted-ca\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157790 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157818 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-bound-sa-token\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157877 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-image-registry-private-configuration\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.157974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.157902 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbqs4\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-kube-api-access-hbqs4\") pod \"926a37d0-25e0-4996-8995-de7bd2115b3c\" (UID: \"926a37d0-25e0-4996-8995-de7bd2115b3c\") " Sep 10 13:33:25.158637 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.158589 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:25.159169 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.159129 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:25.160746 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.160721 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-kube-api-access-hbqs4" (OuterVolumeSpecName: "kube-api-access-hbqs4") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "kube-api-access-hbqs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:25.160841 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.160822 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:33:25.160958 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.160923 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:25.161191 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.161163 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:25.161402 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.161379 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:33:25.168762 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.168739 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926a37d0-25e0-4996-8995-de7bd2115b3c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "926a37d0-25e0-4996-8995-de7bd2115b3c" (UID: "926a37d0-25e0-4996-8995-de7bd2115b3c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:33:25.258756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258720 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-certificates\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.258756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258750 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-installation-pull-secrets\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.258756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258764 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/926a37d0-25e0-4996-8995-de7bd2115b3c-trusted-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.259001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258776 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-registry-tls\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.259001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258787 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-bound-sa-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.259001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258800 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/926a37d0-25e0-4996-8995-de7bd2115b3c-image-registry-private-configuration\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.259001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258811 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbqs4\" (UniqueName: \"kubernetes.io/projected/926a37d0-25e0-4996-8995-de7bd2115b3c-kube-api-access-hbqs4\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.259001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.258823 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/926a37d0-25e0-4996-8995-de7bd2115b3c-ca-trust-extracted\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:33:25.391130 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.391097 2568 generic.go:356] "Generic (PLEG): container finished" podID="926a37d0-25e0-4996-8995-de7bd2115b3c" containerID="b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e" exitCode=0 Sep 10 13:33:25.391301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.391139 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" Sep 10 13:33:25.391301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.391180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" event={"ID":"926a37d0-25e0-4996-8995-de7bd2115b3c","Type":"ContainerDied","Data":"b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e"} Sep 10 13:33:25.391301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.391262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746bc7545d-7r2l4" event={"ID":"926a37d0-25e0-4996-8995-de7bd2115b3c","Type":"ContainerDied","Data":"90603d24cf62a172677916d18e731d30cf59b6f7e9af9ae00a8c907ea95cfd8f"} Sep 10 13:33:25.391301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.391292 2568 scope.go:117] "RemoveContainer" containerID="b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e" Sep 10 13:33:25.392799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.392775 2568 generic.go:356] "Generic (PLEG): container finished" podID="b0865057-5202-4207-a761-5ec236bf4b84" containerID="0f0691d3465756c00e656b6ec5b8bb12f71d6ca35f612645d77389f514d2a26e" exitCode=0 Sep 10 13:33:25.392898 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.392817 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" event={"ID":"b0865057-5202-4207-a761-5ec236bf4b84","Type":"ContainerDied","Data":"0f0691d3465756c00e656b6ec5b8bb12f71d6ca35f612645d77389f514d2a26e"} Sep 10 13:33:25.393168 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.393145 2568 scope.go:117] "RemoveContainer" containerID="0f0691d3465756c00e656b6ec5b8bb12f71d6ca35f612645d77389f514d2a26e" Sep 10 13:33:25.403178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.403156 2568 scope.go:117] "RemoveContainer" containerID="b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e" Sep 10 13:33:25.403453 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:33:25.403433 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e\": container with ID starting with b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e not found: ID does not exist" containerID="b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e" Sep 10 13:33:25.403500 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.403460 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e"} err="failed to get container status \"b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e\": rpc error: code = NotFound desc = could not find container \"b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e\": container with ID starting with b842688ae988fab5d4be20cffbafc8f4c8b0a495cce244b249037c49a6480f5e not found: ID does not exist" Sep 10 13:33:25.440016 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.439970 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-746bc7545d-7r2l4"] Sep 10 13:33:25.442040 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:25.442014 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-746bc7545d-7r2l4"] Sep 10 13:33:26.399035 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:26.399000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7lldn" event={"ID":"b0865057-5202-4207-a761-5ec236bf4b84","Type":"ContainerStarted","Data":"79446d0e6e56ced36b62212a766b332e907f283b086f182e93db54497345bfae"} Sep 10 13:33:26.422705 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:26.422671 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926a37d0-25e0-4996-8995-de7bd2115b3c" path="/var/lib/kubelet/pods/926a37d0-25e0-4996-8995-de7bd2115b3c/volumes" Sep 10 13:33:39.451272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:39.451232 2568 generic.go:356] "Generic (PLEG): container finished" podID="836f29b4-3ff1-47fc-ab08-28dcdff5144c" containerID="a96646e1493f7e305254080423435bb1ccf9625112c087386c8d6d7929c7b34d" exitCode=0 Sep 10 13:33:39.451739 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:39.451293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" event={"ID":"836f29b4-3ff1-47fc-ab08-28dcdff5144c","Type":"ContainerDied","Data":"a96646e1493f7e305254080423435bb1ccf9625112c087386c8d6d7929c7b34d"} Sep 10 13:33:39.451739 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:39.451641 2568 scope.go:117] "RemoveContainer" containerID="a96646e1493f7e305254080423435bb1ccf9625112c087386c8d6d7929c7b34d" Sep 10 13:33:40.461456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:40.461410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-m5lfk" event={"ID":"836f29b4-3ff1-47fc-ab08-28dcdff5144c","Type":"ContainerStarted","Data":"00d4a93d28ef965fd0282f3f795191c6f14181096e39538faf7cd5b074011c2e"} Sep 10 13:33:41.337662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:41.337628 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zwfnn" Sep 10 13:33:42.299974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:42.299943 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:33:42.303292 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:42.303273 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:33:42.307455 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:42.307432 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:33:42.310962 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:42.310939 2568 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 13:33:42.311717 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:33:42.311699 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:36:38.532314 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.532234 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr"] Sep 10 13:36:38.532772 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.532542 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926a37d0-25e0-4996-8995-de7bd2115b3c" containerName="registry" Sep 10 13:36:38.532772 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.532555 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="926a37d0-25e0-4996-8995-de7bd2115b3c" containerName="registry" Sep 10 13:36:38.532772 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.532636 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="926a37d0-25e0-4996-8995-de7bd2115b3c" containerName="registry" Sep 10 13:36:38.535557 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.535541 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.537961 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.537933 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:36:38.538088 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.537979 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m87wf\"" Sep 10 13:36:38.539100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.539082 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:36:38.546807 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.546788 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr"] Sep 10 13:36:38.650001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.649965 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbdlg\" (UniqueName: \"kubernetes.io/projected/5eaeb756-b802-4612-8eb2-c9c96adaa96d-kube-api-access-gbdlg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.650173 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.650009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.650173 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.650029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.750836 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.750805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbdlg\" (UniqueName: \"kubernetes.io/projected/5eaeb756-b802-4612-8eb2-c9c96adaa96d-kube-api-access-gbdlg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.751005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.750853 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.751005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.750876 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.751253 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.751232 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.751317 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.751252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.767242 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.767219 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbdlg\" (UniqueName: \"kubernetes.io/projected/5eaeb756-b802-4612-8eb2-c9c96adaa96d-kube-api-access-gbdlg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:38.844799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:38.844724 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:39.116332 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:39.116309 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr"] Sep 10 13:36:39.118313 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:36:39.118287 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eaeb756_b802_4612_8eb2_c9c96adaa96d.slice/crio-4082226184c132d39e44a55fbe9004e16dd453dd4cbde71478b9650408a61d24 WatchSource:0}: Error finding container 4082226184c132d39e44a55fbe9004e16dd453dd4cbde71478b9650408a61d24: Status 404 returned error can't find the container with id 4082226184c132d39e44a55fbe9004e16dd453dd4cbde71478b9650408a61d24 Sep 10 13:36:39.120129 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:39.120113 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:36:40.109798 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:40.109763 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" event={"ID":"5eaeb756-b802-4612-8eb2-c9c96adaa96d","Type":"ContainerStarted","Data":"4082226184c132d39e44a55fbe9004e16dd453dd4cbde71478b9650408a61d24"} Sep 10 13:36:46.140952 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:46.140864 2568 generic.go:356] "Generic (PLEG): container finished" podID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerID="5b048ee9ee1e8760121d17b8bb17b6d7e1e5961c08e75c06c9aaf1f5347fcca5" exitCode=0 Sep 10 13:36:46.141317 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:46.140955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" event={"ID":"5eaeb756-b802-4612-8eb2-c9c96adaa96d","Type":"ContainerDied","Data":"5b048ee9ee1e8760121d17b8bb17b6d7e1e5961c08e75c06c9aaf1f5347fcca5"} Sep 10 13:36:50.158725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:50.158695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" event={"ID":"5eaeb756-b802-4612-8eb2-c9c96adaa96d","Type":"ContainerStarted","Data":"cdd59228360c3220a1405e75d13ec8b52e87064dc692f9a7bb8050f0fc8f2a0e"} Sep 10 13:36:51.163677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:51.163642 2568 generic.go:356] "Generic (PLEG): container finished" podID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerID="cdd59228360c3220a1405e75d13ec8b52e87064dc692f9a7bb8050f0fc8f2a0e" exitCode=0 Sep 10 13:36:51.164068 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:51.163701 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" event={"ID":"5eaeb756-b802-4612-8eb2-c9c96adaa96d","Type":"ContainerDied","Data":"cdd59228360c3220a1405e75d13ec8b52e87064dc692f9a7bb8050f0fc8f2a0e"} Sep 10 13:36:58.200314 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:58.200276 2568 generic.go:356] "Generic (PLEG): container finished" podID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerID="90500fd8608097b741b852a1ed85d1688534fd24b958b526a304f4701e4a66f8" exitCode=0 Sep 10 13:36:58.200718 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:58.200313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" event={"ID":"5eaeb756-b802-4612-8eb2-c9c96adaa96d","Type":"ContainerDied","Data":"90500fd8608097b741b852a1ed85d1688534fd24b958b526a304f4701e4a66f8"} Sep 10 13:36:59.330078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.330050 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:36:59.520541 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.520498 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbdlg\" (UniqueName: \"kubernetes.io/projected/5eaeb756-b802-4612-8eb2-c9c96adaa96d-kube-api-access-gbdlg\") pod \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " Sep 10 13:36:59.520767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.520578 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-util\") pod \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " Sep 10 13:36:59.520767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.520630 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-bundle\") pod \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\" (UID: \"5eaeb756-b802-4612-8eb2-c9c96adaa96d\") " Sep 10 13:36:59.521425 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.521394 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-bundle" (OuterVolumeSpecName: "bundle") pod "5eaeb756-b802-4612-8eb2-c9c96adaa96d" (UID: "5eaeb756-b802-4612-8eb2-c9c96adaa96d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:59.522916 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.522884 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaeb756-b802-4612-8eb2-c9c96adaa96d-kube-api-access-gbdlg" (OuterVolumeSpecName: "kube-api-access-gbdlg") pod "5eaeb756-b802-4612-8eb2-c9c96adaa96d" (UID: "5eaeb756-b802-4612-8eb2-c9c96adaa96d"). InnerVolumeSpecName "kube-api-access-gbdlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:36:59.525380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.525356 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-util" (OuterVolumeSpecName: "util") pod "5eaeb756-b802-4612-8eb2-c9c96adaa96d" (UID: "5eaeb756-b802-4612-8eb2-c9c96adaa96d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:59.621504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.621458 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbdlg\" (UniqueName: \"kubernetes.io/projected/5eaeb756-b802-4612-8eb2-c9c96adaa96d-kube-api-access-gbdlg\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:36:59.621504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.621493 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-util\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:36:59.621504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:36:59.621502 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaeb756-b802-4612-8eb2-c9c96adaa96d-bundle\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:37:00.209769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:00.209733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" event={"ID":"5eaeb756-b802-4612-8eb2-c9c96adaa96d","Type":"ContainerDied","Data":"4082226184c132d39e44a55fbe9004e16dd453dd4cbde71478b9650408a61d24"} Sep 10 13:37:00.209769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:00.209755 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406x9kr" Sep 10 13:37:00.209967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:00.209786 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4082226184c132d39e44a55fbe9004e16dd453dd4cbde71478b9650408a61d24" Sep 10 13:37:06.643108 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643078 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k"] Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643336 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="pull" Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643346 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="pull" Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643366 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="util" Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643371 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="util" Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643379 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="extract" Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643384 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="extract" Sep 10 13:37:06.643564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.643433 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eaeb756-b802-4612-8eb2-c9c96adaa96d" containerName="extract" Sep 10 13:37:06.646253 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.646237 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.649031 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.649011 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:37:06.649526 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.649510 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ctshm\"" Sep 10 13:37:06.649622 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.649592 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 13:37:06.661699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.661675 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k"] Sep 10 13:37:06.672494 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.672470 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4bt\" (UniqueName: \"kubernetes.io/projected/be28d841-01c8-4bac-8a09-daa6ddc64c90-kube-api-access-6c4bt\") pod \"cert-manager-operator-controller-manager-55c444bf86-6ck2k\" (UID: \"be28d841-01c8-4bac-8a09-daa6ddc64c90\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.672579 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.672517 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be28d841-01c8-4bac-8a09-daa6ddc64c90-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6ck2k\" (UID: \"be28d841-01c8-4bac-8a09-daa6ddc64c90\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.773812 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.773785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4bt\" (UniqueName: \"kubernetes.io/projected/be28d841-01c8-4bac-8a09-daa6ddc64c90-kube-api-access-6c4bt\") pod \"cert-manager-operator-controller-manager-55c444bf86-6ck2k\" (UID: \"be28d841-01c8-4bac-8a09-daa6ddc64c90\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.773924 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.773828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be28d841-01c8-4bac-8a09-daa6ddc64c90-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6ck2k\" (UID: \"be28d841-01c8-4bac-8a09-daa6ddc64c90\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.774137 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.774123 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be28d841-01c8-4bac-8a09-daa6ddc64c90-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6ck2k\" (UID: \"be28d841-01c8-4bac-8a09-daa6ddc64c90\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.783917 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.783891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4bt\" (UniqueName: \"kubernetes.io/projected/be28d841-01c8-4bac-8a09-daa6ddc64c90-kube-api-access-6c4bt\") pod \"cert-manager-operator-controller-manager-55c444bf86-6ck2k\" (UID: \"be28d841-01c8-4bac-8a09-daa6ddc64c90\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:06.955111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:06.955035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" Sep 10 13:37:07.235919 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:07.235897 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k"] Sep 10 13:37:07.237438 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:07.237396 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe28d841_01c8_4bac_8a09_daa6ddc64c90.slice/crio-804230292e77b47e9524cdd8056076e38e1e5232972acba685c2eb32aa1b9313 WatchSource:0}: Error finding container 804230292e77b47e9524cdd8056076e38e1e5232972acba685c2eb32aa1b9313: Status 404 returned error can't find the container with id 804230292e77b47e9524cdd8056076e38e1e5232972acba685c2eb32aa1b9313 Sep 10 13:37:08.239668 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:08.239630 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" event={"ID":"be28d841-01c8-4bac-8a09-daa6ddc64c90","Type":"ContainerStarted","Data":"804230292e77b47e9524cdd8056076e38e1e5232972acba685c2eb32aa1b9313"} Sep 10 13:37:09.245362 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.245329 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" event={"ID":"be28d841-01c8-4bac-8a09-daa6ddc64c90","Type":"ContainerStarted","Data":"c234a6e5f43d0da839369606a3153c2a30da9a712f6bf12212e7a4efaf9c7220"} Sep 10 13:37:09.279156 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.279106 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6ck2k" podStartSLOduration=1.733767954 podStartE2EDuration="3.279090743s" totalImagesPullingTime="1.545322789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:09.276578728 +0000 UTC m=+507.345578704" watchObservedRunningTime="2026-09-10 13:37:09.279090743 +0000 UTC m=+507.348090708" Sep 10 13:37:09.917662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.917626 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t8gx4"] Sep 10 13:37:09.920896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.920880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:09.925169 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.925148 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 13:37:09.926168 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.926152 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gj89m\"" Sep 10 13:37:09.932629 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.932613 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 13:37:09.938357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:09.938335 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t8gx4"] Sep 10 13:37:10.003868 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.003836 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrv4\" (UniqueName: \"kubernetes.io/projected/ed71514a-7d99-483e-b44b-9db6e5de4eee-kube-api-access-mbrv4\") pod \"cert-manager-webhook-6486ffd997-t8gx4\" (UID: \"ed71514a-7d99-483e-b44b-9db6e5de4eee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.004031 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.003876 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed71514a-7d99-483e-b44b-9db6e5de4eee-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t8gx4\" (UID: \"ed71514a-7d99-483e-b44b-9db6e5de4eee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.105218 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.105183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed71514a-7d99-483e-b44b-9db6e5de4eee-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t8gx4\" (UID: \"ed71514a-7d99-483e-b44b-9db6e5de4eee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.105380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.105244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrv4\" (UniqueName: \"kubernetes.io/projected/ed71514a-7d99-483e-b44b-9db6e5de4eee-kube-api-access-mbrv4\") pod \"cert-manager-webhook-6486ffd997-t8gx4\" (UID: \"ed71514a-7d99-483e-b44b-9db6e5de4eee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.114929 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.114908 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrv4\" (UniqueName: \"kubernetes.io/projected/ed71514a-7d99-483e-b44b-9db6e5de4eee-kube-api-access-mbrv4\") pod \"cert-manager-webhook-6486ffd997-t8gx4\" (UID: \"ed71514a-7d99-483e-b44b-9db6e5de4eee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.115253 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.115233 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed71514a-7d99-483e-b44b-9db6e5de4eee-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t8gx4\" (UID: \"ed71514a-7d99-483e-b44b-9db6e5de4eee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.246489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.246466 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:10.521818 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:10.521795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t8gx4"] Sep 10 13:37:10.523928 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:10.523897 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded71514a_7d99_483e_b44b_9db6e5de4eee.slice/crio-4eebbcd5f6b5e48781a5e6c9e1e9619cc3788de12019cdb65be934630bedd89d WatchSource:0}: Error finding container 4eebbcd5f6b5e48781a5e6c9e1e9619cc3788de12019cdb65be934630bedd89d: Status 404 returned error can't find the container with id 4eebbcd5f6b5e48781a5e6c9e1e9619cc3788de12019cdb65be934630bedd89d Sep 10 13:37:11.256981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:11.256944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" event={"ID":"ed71514a-7d99-483e-b44b-9db6e5de4eee","Type":"ContainerStarted","Data":"4eebbcd5f6b5e48781a5e6c9e1e9619cc3788de12019cdb65be934630bedd89d"} Sep 10 13:37:13.268019 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:13.267984 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" event={"ID":"ed71514a-7d99-483e-b44b-9db6e5de4eee","Type":"ContainerStarted","Data":"3c8f7140fdc4bb6b89e108cbfb8b8fa5662b3ae783b6b10fc4779df0b9e23214"} Sep 10 13:37:13.268387 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:13.268038 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:13.290726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:13.290677 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" podStartSLOduration=1.680940635 podStartE2EDuration="4.29066213s" totalImagesPullingTime="2.609721495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:13.289281197 +0000 UTC m=+511.358281162" watchObservedRunningTime="2026-09-10 13:37:13.29066213 +0000 UTC m=+511.359662094" Sep 10 13:37:19.274521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:19.274487 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t8gx4" Sep 10 13:37:20.020040 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.020005 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-szqg6"] Sep 10 13:37:20.023239 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.023223 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.026207 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.026185 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lmsc2\"" Sep 10 13:37:20.037292 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.037268 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-szqg6"] Sep 10 13:37:20.079238 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.079211 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfcr\" (UniqueName: \"kubernetes.io/projected/4045cbf4-a784-4bcf-b547-259fa3a7a3af-kube-api-access-lwfcr\") pod \"cert-manager-55b4d954d5-szqg6\" (UID: \"4045cbf4-a784-4bcf-b547-259fa3a7a3af\") " pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.079345 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.079267 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4045cbf4-a784-4bcf-b547-259fa3a7a3af-bound-sa-token\") pod \"cert-manager-55b4d954d5-szqg6\" (UID: \"4045cbf4-a784-4bcf-b547-259fa3a7a3af\") " pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.180051 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.180029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfcr\" (UniqueName: \"kubernetes.io/projected/4045cbf4-a784-4bcf-b547-259fa3a7a3af-kube-api-access-lwfcr\") pod \"cert-manager-55b4d954d5-szqg6\" (UID: \"4045cbf4-a784-4bcf-b547-259fa3a7a3af\") " pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.180155 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.180082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4045cbf4-a784-4bcf-b547-259fa3a7a3af-bound-sa-token\") pod \"cert-manager-55b4d954d5-szqg6\" (UID: \"4045cbf4-a784-4bcf-b547-259fa3a7a3af\") " pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.191221 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.191197 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfcr\" (UniqueName: \"kubernetes.io/projected/4045cbf4-a784-4bcf-b547-259fa3a7a3af-kube-api-access-lwfcr\") pod \"cert-manager-55b4d954d5-szqg6\" (UID: \"4045cbf4-a784-4bcf-b547-259fa3a7a3af\") " pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.192977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.192956 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4045cbf4-a784-4bcf-b547-259fa3a7a3af-bound-sa-token\") pod \"cert-manager-55b4d954d5-szqg6\" (UID: \"4045cbf4-a784-4bcf-b547-259fa3a7a3af\") " pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.331530 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.331452 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-szqg6" Sep 10 13:37:20.604087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:20.604064 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-szqg6"] Sep 10 13:37:20.606758 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:20.606726 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4045cbf4_a784_4bcf_b547_259fa3a7a3af.slice/crio-a37a2c4fb2143cc8a4091735c82b3bbc21505c6d9c2b14f70a9ce7d069da7fbe WatchSource:0}: Error finding container a37a2c4fb2143cc8a4091735c82b3bbc21505c6d9c2b14f70a9ce7d069da7fbe: Status 404 returned error can't find the container with id a37a2c4fb2143cc8a4091735c82b3bbc21505c6d9c2b14f70a9ce7d069da7fbe Sep 10 13:37:21.301815 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:21.301779 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-szqg6" event={"ID":"4045cbf4-a784-4bcf-b547-259fa3a7a3af","Type":"ContainerStarted","Data":"192300ab74e99a5a4e596ba619322212e8bbcf61547b802d0a39c534d3eb8104"} Sep 10 13:37:21.301977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:21.301825 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-szqg6" event={"ID":"4045cbf4-a784-4bcf-b547-259fa3a7a3af","Type":"ContainerStarted","Data":"a37a2c4fb2143cc8a4091735c82b3bbc21505c6d9c2b14f70a9ce7d069da7fbe"} Sep 10 13:37:21.330165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:21.330118 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-szqg6" podStartSLOduration=2.330099482 podStartE2EDuration="2.330099482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:21.328989815 +0000 UTC m=+519.397989779" watchObservedRunningTime="2026-09-10 13:37:21.330099482 +0000 UTC m=+519.399099447" Sep 10 13:37:41.502658 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.502562 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd"] Sep 10 13:37:41.505907 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.505891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.510545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.510523 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:37:41.511885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.511861 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:37:41.511987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.511905 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m87wf\"" Sep 10 13:37:41.537469 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.537447 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd"] Sep 10 13:37:41.642458 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.642428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.642458 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.642462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988mw\" (UniqueName: \"kubernetes.io/projected/7998c142-e38d-499e-93a2-c0ba4d92a438-kube-api-access-988mw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.642798 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.642496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.743110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.743077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.743273 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.743156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.743273 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.743186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988mw\" (UniqueName: \"kubernetes.io/projected/7998c142-e38d-499e-93a2-c0ba4d92a438-kube-api-access-988mw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.743498 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.743461 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.743558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.743490 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.760493 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.760436 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-988mw\" (UniqueName: \"kubernetes.io/projected/7998c142-e38d-499e-93a2-c0ba4d92a438-kube-api-access-988mw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:41.815481 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:41.815457 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:42.089672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:42.089645 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd"] Sep 10 13:37:42.091037 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:42.091008 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7998c142_e38d_499e_93a2_c0ba4d92a438.slice/crio-903e07c40f2fd10114089937767ba7134f3fff33ec3da897a3cd7292093ac237 WatchSource:0}: Error finding container 903e07c40f2fd10114089937767ba7134f3fff33ec3da897a3cd7292093ac237: Status 404 returned error can't find the container with id 903e07c40f2fd10114089937767ba7134f3fff33ec3da897a3cd7292093ac237 Sep 10 13:37:42.391098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:42.391021 2568 generic.go:356] "Generic (PLEG): container finished" podID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerID="0e1f9b8317bad1d1de7e8229bd8a225ddf915d91b57a22bb5865aed9ff5c8e71" exitCode=0 Sep 10 13:37:42.391098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:42.391065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" event={"ID":"7998c142-e38d-499e-93a2-c0ba4d92a438","Type":"ContainerDied","Data":"0e1f9b8317bad1d1de7e8229bd8a225ddf915d91b57a22bb5865aed9ff5c8e71"} Sep 10 13:37:42.391275 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:42.391105 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" event={"ID":"7998c142-e38d-499e-93a2-c0ba4d92a438","Type":"ContainerStarted","Data":"903e07c40f2fd10114089937767ba7134f3fff33ec3da897a3cd7292093ac237"} Sep 10 13:37:44.401756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:44.401718 2568 generic.go:356] "Generic (PLEG): container finished" podID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerID="1dcbb170581001af08610ab7f67170fcddf1c583e6b17301226004c00af00998" exitCode=0 Sep 10 13:37:44.402118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:44.401785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" event={"ID":"7998c142-e38d-499e-93a2-c0ba4d92a438","Type":"ContainerDied","Data":"1dcbb170581001af08610ab7f67170fcddf1c583e6b17301226004c00af00998"} Sep 10 13:37:45.408307 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.408278 2568 generic.go:356] "Generic (PLEG): container finished" podID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerID="449a0d5660b3dead06a46a18f04743a971994e5bd3ec2bc46bce6cedd7ee1859" exitCode=0 Sep 10 13:37:45.408720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.408321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" event={"ID":"7998c142-e38d-499e-93a2-c0ba4d92a438","Type":"ContainerDied","Data":"449a0d5660b3dead06a46a18f04743a971994e5bd3ec2bc46bce6cedd7ee1859"} Sep 10 13:37:45.705259 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.705174 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g"] Sep 10 13:37:45.708104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.708087 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.712699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.712681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:37:45.713121 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.713094 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 13:37:45.713893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.713874 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 13:37:45.714284 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.714070 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 13:37:45.714284 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.714087 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 13:37:45.714420 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.714348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mjg6w\"" Sep 10 13:37:45.728478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.728456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g"] Sep 10 13:37:45.775378 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.775353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27n6j\" (UniqueName: \"kubernetes.io/projected/0b14eae4-9c67-4080-9662-b0370dfddd41-kube-api-access-27n6j\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.775485 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.775384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b14eae4-9c67-4080-9662-b0370dfddd41-cert\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.775485 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.775441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b14eae4-9c67-4080-9662-b0370dfddd41-manager-config\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.775585 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.775517 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b14eae4-9c67-4080-9662-b0370dfddd41-metrics-cert\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.876825 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.876793 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b14eae4-9c67-4080-9662-b0370dfddd41-manager-config\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.876981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.876843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b14eae4-9c67-4080-9662-b0370dfddd41-metrics-cert\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.876981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.876872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27n6j\" (UniqueName: \"kubernetes.io/projected/0b14eae4-9c67-4080-9662-b0370dfddd41-kube-api-access-27n6j\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.876981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.876892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b14eae4-9c67-4080-9662-b0370dfddd41-cert\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.877521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.877500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b14eae4-9c67-4080-9662-b0370dfddd41-manager-config\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.879352 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.879329 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b14eae4-9c67-4080-9662-b0370dfddd41-cert\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.879445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.879399 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b14eae4-9c67-4080-9662-b0370dfddd41-metrics-cert\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:45.887721 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:45.887696 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27n6j\" (UniqueName: \"kubernetes.io/projected/0b14eae4-9c67-4080-9662-b0370dfddd41-kube-api-access-27n6j\") pod \"lws-controller-manager-7c4cb59d47-knx4g\" (UID: \"0b14eae4-9c67-4080-9662-b0370dfddd41\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:46.016852 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.016824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:46.294539 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:46.294511 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b14eae4_9c67_4080_9662_b0370dfddd41.slice/crio-c796d3fbbfb24da18a3be0c2342c24ca3a45c17419b9546cf66d34fb7c8c6b7c WatchSource:0}: Error finding container c796d3fbbfb24da18a3be0c2342c24ca3a45c17419b9546cf66d34fb7c8c6b7c: Status 404 returned error can't find the container with id c796d3fbbfb24da18a3be0c2342c24ca3a45c17419b9546cf66d34fb7c8c6b7c Sep 10 13:37:46.296232 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.296159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g"] Sep 10 13:37:46.413837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.413804 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" event={"ID":"0b14eae4-9c67-4080-9662-b0370dfddd41","Type":"ContainerStarted","Data":"c796d3fbbfb24da18a3be0c2342c24ca3a45c17419b9546cf66d34fb7c8c6b7c"} Sep 10 13:37:46.525936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.525911 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:46.582515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.582412 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-util\") pod \"7998c142-e38d-499e-93a2-c0ba4d92a438\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " Sep 10 13:37:46.582515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.582487 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-bundle\") pod \"7998c142-e38d-499e-93a2-c0ba4d92a438\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " Sep 10 13:37:46.583025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.582927 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-988mw\" (UniqueName: \"kubernetes.io/projected/7998c142-e38d-499e-93a2-c0ba4d92a438-kube-api-access-988mw\") pod \"7998c142-e38d-499e-93a2-c0ba4d92a438\" (UID: \"7998c142-e38d-499e-93a2-c0ba4d92a438\") " Sep 10 13:37:46.583409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.583366 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-bundle" (OuterVolumeSpecName: "bundle") pod "7998c142-e38d-499e-93a2-c0ba4d92a438" (UID: "7998c142-e38d-499e-93a2-c0ba4d92a438"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:46.585030 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.584984 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7998c142-e38d-499e-93a2-c0ba4d92a438-kube-api-access-988mw" (OuterVolumeSpecName: "kube-api-access-988mw") pod "7998c142-e38d-499e-93a2-c0ba4d92a438" (UID: "7998c142-e38d-499e-93a2-c0ba4d92a438"). InnerVolumeSpecName "kube-api-access-988mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:46.589213 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.589188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-util" (OuterVolumeSpecName: "util") pod "7998c142-e38d-499e-93a2-c0ba4d92a438" (UID: "7998c142-e38d-499e-93a2-c0ba4d92a438"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:46.684141 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.684110 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-bundle\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:37:46.684141 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.684137 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988mw\" (UniqueName: \"kubernetes.io/projected/7998c142-e38d-499e-93a2-c0ba4d92a438-kube-api-access-988mw\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:37:46.684141 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:46.684147 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7998c142-e38d-499e-93a2-c0ba4d92a438-util\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:37:47.422027 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:47.421999 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" Sep 10 13:37:47.422496 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:47.421994 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdp7gd" event={"ID":"7998c142-e38d-499e-93a2-c0ba4d92a438","Type":"ContainerDied","Data":"903e07c40f2fd10114089937767ba7134f3fff33ec3da897a3cd7292093ac237"} Sep 10 13:37:47.422496 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:47.422112 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="903e07c40f2fd10114089937767ba7134f3fff33ec3da897a3cd7292093ac237" Sep 10 13:37:49.433214 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:49.433180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" event={"ID":"0b14eae4-9c67-4080-9662-b0370dfddd41","Type":"ContainerStarted","Data":"e3d522b5c5ea5703a3d7473fb5956bca8b613b4ad65f73c41e3e343ea2999a2e"} Sep 10 13:37:49.433589 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:49.433260 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:37:49.465175 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:49.465127 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" podStartSLOduration=2.361630898 podStartE2EDuration="4.465113104s" totalImagesPullingTime="2.103482206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:49.463185915 +0000 UTC m=+547.532185879" watchObservedRunningTime="2026-09-10 13:37:49.465113104 +0000 UTC m=+547.534113068" Sep 10 13:37:58.409738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.409704 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj"] Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410003 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="pull" Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410017 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="pull" Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410041 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="util" Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410047 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="util" Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410053 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="extract" Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410058 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="extract" Sep 10 13:37:58.410111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.410110 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7998c142-e38d-499e-93a2-c0ba4d92a438" containerName="extract" Sep 10 13:37:58.415693 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.414463 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.419416 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.419395 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:37:58.419751 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.419727 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:37:58.419853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.419760 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m87wf\"" Sep 10 13:37:58.425408 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.425387 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj"] Sep 10 13:37:58.577181 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.577151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.577379 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.577206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nv9l\" (UniqueName: \"kubernetes.io/projected/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-kube-api-access-2nv9l\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.577379 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.577247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.678286 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.678207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nv9l\" (UniqueName: \"kubernetes.io/projected/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-kube-api-access-2nv9l\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.678286 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.678252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.678465 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.678297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.678719 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.678697 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.678719 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.678709 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.687836 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.687813 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nv9l\" (UniqueName: \"kubernetes.io/projected/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-kube-api-access-2nv9l\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.724747 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.724726 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:37:58.852327 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.852297 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56"] Sep 10 13:37:58.856845 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.856825 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:58.861823 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.861804 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6c66s\"" Sep 10 13:37:58.861921 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.861805 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 13:37:58.862053 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.862041 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 13:37:58.862346 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.862327 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 13:37:58.862912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.862896 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 13:37:58.884998 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.884969 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56"] Sep 10 13:37:58.980565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.980538 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:58.980729 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.980584 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpc89\" (UniqueName: \"kubernetes.io/projected/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-kube-api-access-vpc89\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:58.980729 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.980642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:58.980729 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.980683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:58.980729 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:58.980701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.017933 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.017857 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj"] Sep 10 13:37:59.020120 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:59.020093 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2aa8831_aa47_48b2_95ae_35bc3a64fd5f.slice/crio-16f577cc929c4e2e3c36236fed4c772a695bf18e89a193e9eba54c579917c35f WatchSource:0}: Error finding container 16f577cc929c4e2e3c36236fed4c772a695bf18e89a193e9eba54c579917c35f: Status 404 returned error can't find the container with id 16f577cc929c4e2e3c36236fed4c772a695bf18e89a193e9eba54c579917c35f Sep 10 13:37:59.082015 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.081987 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.082118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.082034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpc89\" (UniqueName: \"kubernetes.io/projected/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-kube-api-access-vpc89\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.082118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.082056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.082118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.082084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.082118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.082102 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.082487 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.082464 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.084076 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.084055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.084406 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.084390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.084693 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.084669 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.102023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.101991 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpc89\" (UniqueName: \"kubernetes.io/projected/6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93-kube-api-access-vpc89\") pod \"opendatahub-operator-controller-manager-579d56558b-94r56\" (UID: \"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.166789 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.166766 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:37:59.459229 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.459202 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56"] Sep 10 13:37:59.460462 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:37:59.460438 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e101e8e_0c24_4ee3_83a8_58f0ca2e6f93.slice/crio-12be1b05a85fd3519e4b596f17efed95a314724e7b5e9f805944fe644446b2b5 WatchSource:0}: Error finding container 12be1b05a85fd3519e4b596f17efed95a314724e7b5e9f805944fe644446b2b5: Status 404 returned error can't find the container with id 12be1b05a85fd3519e4b596f17efed95a314724e7b5e9f805944fe644446b2b5 Sep 10 13:37:59.480144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.480115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" event={"ID":"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93","Type":"ContainerStarted","Data":"12be1b05a85fd3519e4b596f17efed95a314724e7b5e9f805944fe644446b2b5"} Sep 10 13:37:59.481565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.481533 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerID="54b5f1935d53389f1eadef9708bde0b1a379c11655074c87235588b017d17350" exitCode=0 Sep 10 13:37:59.481677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.481624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" event={"ID":"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f","Type":"ContainerDied","Data":"54b5f1935d53389f1eadef9708bde0b1a379c11655074c87235588b017d17350"} Sep 10 13:37:59.481677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:37:59.481667 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" event={"ID":"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f","Type":"ContainerStarted","Data":"16f577cc929c4e2e3c36236fed4c772a695bf18e89a193e9eba54c579917c35f"} Sep 10 13:38:00.440389 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:00.440313 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-knx4g" Sep 10 13:38:00.492464 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:00.492306 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerID="5dc8c15eccc482770dd7576dc40c24f11e6eecb6f02a284d72d5500bae08f468" exitCode=0 Sep 10 13:38:00.492464 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:00.492399 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" event={"ID":"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f","Type":"ContainerDied","Data":"5dc8c15eccc482770dd7576dc40c24f11e6eecb6f02a284d72d5500bae08f468"} Sep 10 13:38:01.500508 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:01.500471 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerID="453e86aa44e60356bdcd16e3a062785d97bbfe83a1ad28a7284b1e526f4b3428" exitCode=0 Sep 10 13:38:01.500903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:01.500547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" event={"ID":"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f","Type":"ContainerDied","Data":"453e86aa44e60356bdcd16e3a062785d97bbfe83a1ad28a7284b1e526f4b3428"} Sep 10 13:38:02.626712 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.626689 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:38:02.716380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.716350 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-util\") pod \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " Sep 10 13:38:02.716534 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.716396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nv9l\" (UniqueName: \"kubernetes.io/projected/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-kube-api-access-2nv9l\") pod \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " Sep 10 13:38:02.716534 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.716419 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-bundle\") pod \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\" (UID: \"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f\") " Sep 10 13:38:02.717193 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.717169 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-bundle" (OuterVolumeSpecName: "bundle") pod "a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" (UID: "a2aa8831-aa47-48b2-95ae-35bc3a64fd5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:38:02.718511 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.718486 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-kube-api-access-2nv9l" (OuterVolumeSpecName: "kube-api-access-2nv9l") pod "a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" (UID: "a2aa8831-aa47-48b2-95ae-35bc3a64fd5f"). InnerVolumeSpecName "kube-api-access-2nv9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:38:02.721388 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.721357 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-util" (OuterVolumeSpecName: "util") pod "a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" (UID: "a2aa8831-aa47-48b2-95ae-35bc3a64fd5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:38:02.817262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.817220 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-util\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:38:02.817262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.817255 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nv9l\" (UniqueName: \"kubernetes.io/projected/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-kube-api-access-2nv9l\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:38:02.817262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:02.817264 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2aa8831-aa47-48b2-95ae-35bc3a64fd5f-bundle\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:38:03.511106 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:03.511064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" event={"ID":"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93","Type":"ContainerStarted","Data":"6643cb6e7c22f724e3986a9bfda01e5ab5d434db44335bb56d258e9e6da77f44"} Sep 10 13:38:03.512911 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:03.512889 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" Sep 10 13:38:03.512911 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:03.512897 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dlbqj" event={"ID":"a2aa8831-aa47-48b2-95ae-35bc3a64fd5f","Type":"ContainerDied","Data":"16f577cc929c4e2e3c36236fed4c772a695bf18e89a193e9eba54c579917c35f"} Sep 10 13:38:03.513121 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:03.512935 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f577cc929c4e2e3c36236fed4c772a695bf18e89a193e9eba54c579917c35f" Sep 10 13:38:04.519281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:04.519248 2568 generic.go:356] "Generic (PLEG): container finished" podID="6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93" containerID="6643cb6e7c22f724e3986a9bfda01e5ab5d434db44335bb56d258e9e6da77f44" exitCode=0 Sep 10 13:38:04.519772 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:04.519290 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" event={"ID":"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93","Type":"ContainerDied","Data":"6643cb6e7c22f724e3986a9bfda01e5ab5d434db44335bb56d258e9e6da77f44"} Sep 10 13:38:05.525616 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:05.525561 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" event={"ID":"6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93","Type":"ContainerStarted","Data":"0e60535c6886b8de59d94ccb13150cc0d2683974d1e8ffbbcb7b6374fd9533fd"} Sep 10 13:38:05.526059 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:05.525668 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:38:05.554339 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:05.554290 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" podStartSLOduration=3.123301157 podStartE2EDuration="7.554274401s" totalImagesPullingTime="3.315799485s" totalInitContainerRuntime="1.115173759s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:05.552427115 +0000 UTC m=+563.621427078" watchObservedRunningTime="2026-09-10 13:38:05.554274401 +0000 UTC m=+563.623274365" Sep 10 13:38:16.532500 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:16.532467 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-94r56" Sep 10 13:38:26.277931 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.277898 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4"] Sep 10 13:38:26.278287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278218 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="extract" Sep 10 13:38:26.278287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278230 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="extract" Sep 10 13:38:26.278287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278240 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="pull" Sep 10 13:38:26.278287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278247 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="pull" Sep 10 13:38:26.278287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278260 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="util" Sep 10 13:38:26.278287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278265 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="util" Sep 10 13:38:26.278478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.278313 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2aa8831-aa47-48b2-95ae-35bc3a64fd5f" containerName="extract" Sep 10 13:38:26.292769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.292749 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.297192 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.297171 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4"] Sep 10 13:38:26.302176 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.302155 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-j76jg\"" Sep 10 13:38:26.318796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.318775 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 13:38:26.388815 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.388783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.388939 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.388825 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.388939 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.388851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.388939 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.388926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5v49\" (UniqueName: \"kubernetes.io/projected/7ca44212-723b-43e1-a744-e64bbd11ed44-kube-api-access-j5v49\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.389072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.388955 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.389072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.388985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.389072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.389005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.389072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.389041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.389072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.389071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca44212-723b-43e1-a744-e64bbd11ed44-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.489797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.489761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca44212-723b-43e1-a744-e64bbd11ed44-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.489977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.489814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.489977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.489929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490103 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.489980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490103 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490046 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5v49\" (UniqueName: \"kubernetes.io/projected/7ca44212-723b-43e1-a744-e64bbd11ed44-kube-api-access-j5v49\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490103 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490255 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490255 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490169 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490255 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490255 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490527 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490508 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490622 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490540 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490622 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.490622 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.490546 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca44212-723b-43e1-a744-e64bbd11ed44-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.492374 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.492344 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.492862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.492841 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.501528 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.501500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5v49\" (UniqueName: \"kubernetes.io/projected/7ca44212-723b-43e1-a744-e64bbd11ed44-kube-api-access-j5v49\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.506427 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.506402 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca44212-723b-43e1-a744-e64bbd11ed44-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4\" (UID: \"7ca44212-723b-43e1-a744-e64bbd11ed44\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.603870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.603803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:26.885478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:26.885453 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4"] Sep 10 13:38:26.886855 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:38:26.886831 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca44212_723b_43e1_a744_e64bbd11ed44.slice/crio-63f3f2b62e2cc3353708526370ca5dfb2606114ff3205dc569f13a6fd9160660 WatchSource:0}: Error finding container 63f3f2b62e2cc3353708526370ca5dfb2606114ff3205dc569f13a6fd9160660: Status 404 returned error can't find the container with id 63f3f2b62e2cc3353708526370ca5dfb2606114ff3205dc569f13a6fd9160660 Sep 10 13:38:27.625860 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:27.625813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" event={"ID":"7ca44212-723b-43e1-a744-e64bbd11ed44","Type":"ContainerStarted","Data":"63f3f2b62e2cc3353708526370ca5dfb2606114ff3205dc569f13a6fd9160660"} Sep 10 13:38:29.429364 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.429333 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gdssn"] Sep 10 13:38:29.432646 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.432626 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:29.436419 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.436397 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 13:38:29.436518 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.436449 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rtwrm\"" Sep 10 13:38:29.454029 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.454004 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gdssn"] Sep 10 13:38:29.494560 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.494528 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:38:29.494669 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.494589 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:38:29.494669 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.494632 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:38:29.516752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.516731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93709e4-ccec-4ec7-9f0c-777b6c32e703-cert\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:29.516867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.516758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx5xr\" (UniqueName: \"kubernetes.io/projected/d93709e4-ccec-4ec7-9f0c-777b6c32e703-kube-api-access-fx5xr\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:29.617309 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.617272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93709e4-ccec-4ec7-9f0c-777b6c32e703-cert\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:29.617309 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.617309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx5xr\" (UniqueName: \"kubernetes.io/projected/d93709e4-ccec-4ec7-9f0c-777b6c32e703-kube-api-access-fx5xr\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:29.617568 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:38:29.617437 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 13:38:29.617568 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:38:29.617513 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93709e4-ccec-4ec7-9f0c-777b6c32e703-cert podName:d93709e4-ccec-4ec7-9f0c-777b6c32e703 nodeName:}" failed. No retries permitted until 2026-09-10 13:38:30.117491366 +0000 UTC m=+588.186491315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d93709e4-ccec-4ec7-9f0c-777b6c32e703-cert") pod "odh-model-controller-7998c67dc4-gdssn" (UID: "d93709e4-ccec-4ec7-9f0c-777b6c32e703") : secret "odh-model-controller-webhook-cert" not found Sep 10 13:38:29.636904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.636874 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx5xr\" (UniqueName: \"kubernetes.io/projected/d93709e4-ccec-4ec7-9f0c-777b6c32e703-kube-api-access-fx5xr\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:29.638184 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:29.638149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" event={"ID":"7ca44212-723b-43e1-a744-e64bbd11ed44","Type":"ContainerStarted","Data":"f1fa6bb111403d318326a5fe93b89a267c3e54fbed072972237e13553d9c00b0"} Sep 10 13:38:30.120681 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.120648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93709e4-ccec-4ec7-9f0c-777b6c32e703-cert\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:30.122967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.122949 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93709e4-ccec-4ec7-9f0c-777b6c32e703-cert\") pod \"odh-model-controller-7998c67dc4-gdssn\" (UID: \"d93709e4-ccec-4ec7-9f0c-777b6c32e703\") " pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:30.253809 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.253747 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" podStartSLOduration=1.64828406 podStartE2EDuration="4.253725749s" totalImagesPullingTime="2.605441689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:29.680123473 +0000 UTC m=+587.749123439" watchObservedRunningTime="2026-09-10 13:38:30.253725749 +0000 UTC m=+588.322725709" Sep 10 13:38:30.254221 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.254206 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z88b8"] Sep 10 13:38:30.258045 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.258026 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.260645 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.260622 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 13:38:30.268160 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.268141 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z88b8"] Sep 10 13:38:30.322337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.322307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8948k\" (UniqueName: \"kubernetes.io/projected/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-kube-api-access-8948k\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.322455 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.322358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-tls-certs\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.342968 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.342949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:30.422832 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.422781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-tls-certs\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.423001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.422923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8948k\" (UniqueName: \"kubernetes.io/projected/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-kube-api-access-8948k\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.423001 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:38:30.422939 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 13:38:30.423110 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:38:30.423005 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-tls-certs podName:cb29e624-fbf2-4f5d-a69d-5c1a1c16d648 nodeName:}" failed. No retries permitted until 2026-09-10 13:38:30.922985045 +0000 UTC m=+588.991985001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-tls-certs") pod "model-serving-api-54d89bd5fc-z88b8" (UID: "cb29e624-fbf2-4f5d-a69d-5c1a1c16d648") : secret "model-serving-api-tls" not found Sep 10 13:38:30.439698 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.439678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8948k\" (UniqueName: \"kubernetes.io/projected/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-kube-api-access-8948k\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.604074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.604047 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:30.608503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.608479 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:30.632518 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.632497 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gdssn"] Sep 10 13:38:30.635667 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:38:30.635635 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93709e4_ccec_4ec7_9f0c_777b6c32e703.slice/crio-84cc5b68f284d572c7b36183a3df4784aba54884cb9d37d9d78dacf074e8e62c WatchSource:0}: Error finding container 84cc5b68f284d572c7b36183a3df4784aba54884cb9d37d9d78dacf074e8e62c: Status 404 returned error can't find the container with id 84cc5b68f284d572c7b36183a3df4784aba54884cb9d37d9d78dacf074e8e62c Sep 10 13:38:30.643560 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.643536 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" event={"ID":"d93709e4-ccec-4ec7-9f0c-777b6c32e703","Type":"ContainerStarted","Data":"84cc5b68f284d572c7b36183a3df4784aba54884cb9d37d9d78dacf074e8e62c"} Sep 10 13:38:30.643799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.643785 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:30.644489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.644473 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4" Sep 10 13:38:30.928249 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.928170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-tls-certs\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:30.930495 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:30.930474 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb29e624-fbf2-4f5d-a69d-5c1a1c16d648-tls-certs\") pod \"model-serving-api-54d89bd5fc-z88b8\" (UID: \"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648\") " pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:31.168638 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:31.168593 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:31.451787 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:31.451566 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z88b8"] Sep 10 13:38:31.454867 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:38:31.454832 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb29e624_fbf2_4f5d_a69d_5c1a1c16d648.slice/crio-2b081babfd576dfe392b36961a196f83783649c494c67c9ec8f8a3129d974832 WatchSource:0}: Error finding container 2b081babfd576dfe392b36961a196f83783649c494c67c9ec8f8a3129d974832: Status 404 returned error can't find the container with id 2b081babfd576dfe392b36961a196f83783649c494c67c9ec8f8a3129d974832 Sep 10 13:38:31.650858 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:31.650819 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" event={"ID":"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648","Type":"ContainerStarted","Data":"2b081babfd576dfe392b36961a196f83783649c494c67c9ec8f8a3129d974832"} Sep 10 13:38:35.672695 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:35.672652 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" event={"ID":"cb29e624-fbf2-4f5d-a69d-5c1a1c16d648","Type":"ContainerStarted","Data":"9ffd5dbe5f829bb6e902c01d517f607f795532e58ad07b39a2aaf234a5bd58af"} Sep 10 13:38:35.673118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:35.672763 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:38:35.674192 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:35.674169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" event={"ID":"d93709e4-ccec-4ec7-9f0c-777b6c32e703","Type":"ContainerStarted","Data":"a3648398b0379129e1b92f2ea4952618e13da21e7e0b3cea27fecd4c1526367e"} Sep 10 13:38:35.674300 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:35.674247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:35.794841 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:35.794786 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" podStartSLOduration=2.617841934 podStartE2EDuration="5.794770679s" totalImagesPullingTime="3.176928745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:35.788393793 +0000 UTC m=+593.857393756" watchObservedRunningTime="2026-09-10 13:38:35.794770679 +0000 UTC m=+593.863770643" Sep 10 13:38:35.858796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:35.858745 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" podStartSLOduration=2.900424924 podStartE2EDuration="6.858730227s" totalImagesPullingTime="3.958305303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:35.858410825 +0000 UTC m=+593.927410789" watchObservedRunningTime="2026-09-10 13:38:35.858730227 +0000 UTC m=+593.927730189" Sep 10 13:38:38.192558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.192522 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc"] Sep 10 13:38:38.195787 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.195772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.203855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.203830 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 13:38:38.203972 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.203833 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 13:38:38.203972 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.203863 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 13:38:38.204056 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.203979 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7xz5c\"" Sep 10 13:38:38.260483 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.260456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc"] Sep 10 13:38:38.295247 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295218 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7934da2-9826-4123-b961-bbd86a0b1e4a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.295371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.295371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.295515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlp8\" (UniqueName: \"kubernetes.io/projected/b7934da2-9826-4123-b961-bbd86a0b1e4a-kube-api-access-8nlp8\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.295515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295396 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.295515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295417 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.295515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.295431 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.396826 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396800 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.396967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlp8\" (UniqueName: \"kubernetes.io/projected/b7934da2-9826-4123-b961-bbd86a0b1e4a-kube-api-access-8nlp8\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.396967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.396967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.396967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.396967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7934da2-9826-4123-b961-bbd86a0b1e4a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.397303 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.396978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.397648 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.397619 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.399376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.399349 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.399493 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.399414 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.399493 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.399484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.399628 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.399491 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7934da2-9826-4123-b961-bbd86a0b1e4a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.463410 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.463333 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlp8\" (UniqueName: \"kubernetes.io/projected/b7934da2-9826-4123-b961-bbd86a0b1e4a-kube-api-access-8nlp8\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.498851 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.498821 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7934da2-9826-4123-b961-bbd86a0b1e4a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9dzjc\" (UID: \"b7934da2-9826-4123-b961-bbd86a0b1e4a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.504032 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.504010 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:38.856073 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:38.856039 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc"] Sep 10 13:38:38.859560 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:38:38.859534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7934da2_9826_4123_b961_bbd86a0b1e4a.slice/crio-5bb304c8b4063e679027ef4ac572970303a1344f403a40414e336136f7118d21 WatchSource:0}: Error finding container 5bb304c8b4063e679027ef4ac572970303a1344f403a40414e336136f7118d21: Status 404 returned error can't find the container with id 5bb304c8b4063e679027ef4ac572970303a1344f403a40414e336136f7118d21 Sep 10 13:38:39.693769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:39.693731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" event={"ID":"b7934da2-9826-4123-b961-bbd86a0b1e4a","Type":"ContainerStarted","Data":"5bb304c8b4063e679027ef4ac572970303a1344f403a40414e336136f7118d21"} Sep 10 13:38:40.987591 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:40.987551 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:38:40.987875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:40.987658 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:38:41.705247 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:41.705211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" event={"ID":"b7934da2-9826-4123-b961-bbd86a0b1e4a","Type":"ContainerStarted","Data":"f82a7703aee8d69a3e466feb810bfff2109d07725405cf51d7ef9b418afac452"} Sep 10 13:38:41.705425 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:41.705351 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:41.829205 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:41.829152 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" podStartSLOduration=1.703549349 podStartE2EDuration="3.829136762s" totalImagesPullingTime="2.125587413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:41.826254209 +0000 UTC m=+599.895254182" watchObservedRunningTime="2026-09-10 13:38:41.829136762 +0000 UTC m=+599.898136725" Sep 10 13:38:42.340397 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:42.340360 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:38:42.340837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:42.340711 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:38:42.347803 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:42.347784 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:38:42.347954 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:42.347840 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:38:42.711802 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:42.711728 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9dzjc" Sep 10 13:38:46.680987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:46.680956 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gdssn" Sep 10 13:38:46.683104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:38:46.683083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-z88b8" Sep 10 13:41:20.189921 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.189887 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6477b956f8-k6r2p"] Sep 10 13:41:20.193175 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.193156 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.217649 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:20.217608 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-metrics-tls\" is forbidden: User \"system:node:ip-10-0-139-172.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" type="*v1.Secret" Sep 10 13:41:20.219038 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:20.219008 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-controller-6477b956f8-k6r2p\" is forbidden: User \"system:node:ip-10-0-139-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-172.ec2.internal' and this object" podUID="3cccf690-7f60-4b03-8f1b-795c05921fc0" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.219634 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:20.219610 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-139-172.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" Sep 10 13:41:20.219947 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:20.219930 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"maas-parameters\" is forbidden: User \"system:node:ip-10-0-139-172.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-parameters\"" type="*v1.ConfigMap" Sep 10 13:41:20.219947 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:20.219933 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-dockercfg-nn6pf\" is forbidden: User \"system:node:ip-10-0-139-172.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nn6pf\"" type="*v1.Secret" Sep 10 13:41:20.239447 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.239418 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6477b956f8-k6r2p"] Sep 10 13:41:20.259430 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.259399 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rssr\" (UniqueName: \"kubernetes.io/projected/3cccf690-7f60-4b03-8f1b-795c05921fc0-kube-api-access-7rssr\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.259544 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.259443 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-webhook-cert\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.259544 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.259472 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.360090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.360053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rssr\" (UniqueName: \"kubernetes.io/projected/3cccf690-7f60-4b03-8f1b-795c05921fc0-kube-api-access-7rssr\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.360090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.360096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-webhook-cert\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.360369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.360121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:20.381832 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:20.381797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rssr\" (UniqueName: \"kubernetes.io/projected/3cccf690-7f60-4b03-8f1b-795c05921fc0-kube-api-access-7rssr\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:21.219911 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.219878 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 13:41:21.223152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.223132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-webhook-cert\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:21.361215 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:21.361186 2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: failed to sync secret cache: timed out waiting for the condition Sep 10 13:41:21.361374 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:21.361265 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs podName:3cccf690-7f60-4b03-8f1b-795c05921fc0 nodeName:}" failed. No retries permitted until 2026-09-10 13:41:21.861244518 +0000 UTC m=+759.930244461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs") pod "maas-controller-6477b956f8-k6r2p" (UID: "3cccf690-7f60-4b03-8f1b-795c05921fc0") : failed to sync secret cache: timed out waiting for the condition Sep 10 13:41:21.495124 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.495054 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 13:41:21.563846 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.563820 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 13:41:21.804744 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.804671 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nn6pf\"" Sep 10 13:41:21.874170 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.874144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:21.876500 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:21.876470 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs\") pod \"maas-controller-6477b956f8-k6r2p\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:22.003564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:22.003538 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:22.286609 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:22.286573 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6477b956f8-k6r2p"] Sep 10 13:41:22.288750 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:41:22.288726 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cccf690_7f60_4b03_8f1b_795c05921fc0.slice/crio-ab6a42851392ac61468fb46d1bc944478772aba3e60f0d3a4046d86d08224564 WatchSource:0}: Error finding container ab6a42851392ac61468fb46d1bc944478772aba3e60f0d3a4046d86d08224564: Status 404 returned error can't find the container with id ab6a42851392ac61468fb46d1bc944478772aba3e60f0d3a4046d86d08224564 Sep 10 13:41:22.452762 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:22.452725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6477b956f8-k6r2p" event={"ID":"3cccf690-7f60-4b03-8f1b-795c05921fc0","Type":"ContainerStarted","Data":"ab6a42851392ac61468fb46d1bc944478772aba3e60f0d3a4046d86d08224564"} Sep 10 13:41:25.474981 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:25.474937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6477b956f8-k6r2p" event={"ID":"3cccf690-7f60-4b03-8f1b-795c05921fc0","Type":"ContainerStarted","Data":"530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d"} Sep 10 13:41:25.475443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:25.475022 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:34.816266 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:34.816193 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6477b956f8-k6r2p" podStartSLOduration=12.582243502 podStartE2EDuration="14.816169645s" totalImagesPullingTime="2.233926143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:25.517075745 +0000 UTC m=+763.586075705" watchObservedRunningTime="2026-09-10 13:41:34.816169645 +0000 UTC m=+772.885169610" Sep 10 13:41:34.816650 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:34.816397 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6477b956f8-k6r2p"] Sep 10 13:41:34.816726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:34.816644 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6477b956f8-k6r2p" podUID="3cccf690-7f60-4b03-8f1b-795c05921fc0" containerName="manager" containerID="cri-o://530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d" gracePeriod=30 Sep 10 13:41:34.818424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:34.818404 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:35.052864 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.052839 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:35.186635 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.186545 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-webhook-cert\") pod \"3cccf690-7f60-4b03-8f1b-795c05921fc0\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " Sep 10 13:41:35.186635 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.186581 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs\") pod \"3cccf690-7f60-4b03-8f1b-795c05921fc0\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " Sep 10 13:41:35.186804 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.186674 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rssr\" (UniqueName: \"kubernetes.io/projected/3cccf690-7f60-4b03-8f1b-795c05921fc0-kube-api-access-7rssr\") pod \"3cccf690-7f60-4b03-8f1b-795c05921fc0\" (UID: \"3cccf690-7f60-4b03-8f1b-795c05921fc0\") " Sep 10 13:41:35.188572 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.188539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3cccf690-7f60-4b03-8f1b-795c05921fc0" (UID: "3cccf690-7f60-4b03-8f1b-795c05921fc0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:41:35.188752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.188573 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cccf690-7f60-4b03-8f1b-795c05921fc0-kube-api-access-7rssr" (OuterVolumeSpecName: "kube-api-access-7rssr") pod "3cccf690-7f60-4b03-8f1b-795c05921fc0" (UID: "3cccf690-7f60-4b03-8f1b-795c05921fc0"). InnerVolumeSpecName "kube-api-access-7rssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:41:35.188752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.188653 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cccf690-7f60-4b03-8f1b-795c05921fc0" (UID: "3cccf690-7f60-4b03-8f1b-795c05921fc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:41:35.288137 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.288110 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rssr\" (UniqueName: \"kubernetes.io/projected/3cccf690-7f60-4b03-8f1b-795c05921fc0-kube-api-access-7rssr\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:41:35.288137 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.288132 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-webhook-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:41:35.288137 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.288141 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cccf690-7f60-4b03-8f1b-795c05921fc0-metrics-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:41:35.522901 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.522858 2568 generic.go:356] "Generic (PLEG): container finished" podID="3cccf690-7f60-4b03-8f1b-795c05921fc0" containerID="530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d" exitCode=0 Sep 10 13:41:35.523064 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.522911 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6477b956f8-k6r2p" Sep 10 13:41:35.523064 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.522945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6477b956f8-k6r2p" event={"ID":"3cccf690-7f60-4b03-8f1b-795c05921fc0","Type":"ContainerDied","Data":"530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d"} Sep 10 13:41:35.523064 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.522999 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6477b956f8-k6r2p" event={"ID":"3cccf690-7f60-4b03-8f1b-795c05921fc0","Type":"ContainerDied","Data":"ab6a42851392ac61468fb46d1bc944478772aba3e60f0d3a4046d86d08224564"} Sep 10 13:41:35.523064 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.523021 2568 scope.go:117] "RemoveContainer" containerID="530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d" Sep 10 13:41:35.531848 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.531833 2568 scope.go:117] "RemoveContainer" containerID="530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d" Sep 10 13:41:35.532114 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:41:35.532094 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d\": container with ID starting with 530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d not found: ID does not exist" containerID="530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d" Sep 10 13:41:35.532172 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.532120 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d"} err="failed to get container status \"530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d\": rpc error: code = NotFound desc = could not find container \"530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d\": container with ID starting with 530a2598416c9f1a23ba4b121a82e594bd1d610304e6f2792551481e47c6a19d not found: ID does not exist" Sep 10 13:41:35.562424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.562401 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6477b956f8-k6r2p"] Sep 10 13:41:35.566897 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:35.566876 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6477b956f8-k6r2p"] Sep 10 13:41:36.424177 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:36.424143 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cccf690-7f60-4b03-8f1b-795c05921fc0" path="/var/lib/kubelet/pods/3cccf690-7f60-4b03-8f1b-795c05921fc0/volumes" Sep 10 13:41:57.109157 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.109121 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mbq7f"] Sep 10 13:41:57.109726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.109587 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cccf690-7f60-4b03-8f1b-795c05921fc0" containerName="manager" Sep 10 13:41:57.109726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.109623 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cccf690-7f60-4b03-8f1b-795c05921fc0" containerName="manager" Sep 10 13:41:57.109854 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.109753 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cccf690-7f60-4b03-8f1b-795c05921fc0" containerName="manager" Sep 10 13:41:57.113848 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.113815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.116356 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.116330 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 13:41:57.116467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.116369 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8b77f\"" Sep 10 13:41:57.126093 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.126070 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mbq7f"] Sep 10 13:41:57.270754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.270723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be089e-0c86-496e-82b2-592718142acb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mbq7f\" (UID: \"e1be089e-0c86-496e-82b2-592718142acb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.270933 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.270764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7nw\" (UniqueName: \"kubernetes.io/projected/e1be089e-0c86-496e-82b2-592718142acb-kube-api-access-cw7nw\") pod \"payload-processing-5b54b7fbb7-mbq7f\" (UID: \"e1be089e-0c86-496e-82b2-592718142acb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.371969 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.371881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be089e-0c86-496e-82b2-592718142acb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mbq7f\" (UID: \"e1be089e-0c86-496e-82b2-592718142acb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.371969 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.371931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7nw\" (UniqueName: \"kubernetes.io/projected/e1be089e-0c86-496e-82b2-592718142acb-kube-api-access-cw7nw\") pod \"payload-processing-5b54b7fbb7-mbq7f\" (UID: \"e1be089e-0c86-496e-82b2-592718142acb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.372726 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.372706 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be089e-0c86-496e-82b2-592718142acb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mbq7f\" (UID: \"e1be089e-0c86-496e-82b2-592718142acb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.381910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.381892 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7nw\" (UniqueName: \"kubernetes.io/projected/e1be089e-0c86-496e-82b2-592718142acb-kube-api-access-cw7nw\") pod \"payload-processing-5b54b7fbb7-mbq7f\" (UID: \"e1be089e-0c86-496e-82b2-592718142acb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.423297 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.423268 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:57.696437 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.696414 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mbq7f"] Sep 10 13:41:57.698909 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:41:57.698877 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1be089e_0c86_496e_82b2_592718142acb.slice/crio-07a4e6983190173176695ab67236c85f8794bcf9e486ee21f9329771988c3cd3 WatchSource:0}: Error finding container 07a4e6983190173176695ab67236c85f8794bcf9e486ee21f9329771988c3cd3: Status 404 returned error can't find the container with id 07a4e6983190173176695ab67236c85f8794bcf9e486ee21f9329771988c3cd3 Sep 10 13:41:57.701188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:57.701171 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:41:58.638464 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:58.638433 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" event={"ID":"e1be089e-0c86-496e-82b2-592718142acb","Type":"ContainerStarted","Data":"07a4e6983190173176695ab67236c85f8794bcf9e486ee21f9329771988c3cd3"} Sep 10 13:41:59.644714 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:59.644624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" event={"ID":"e1be089e-0c86-496e-82b2-592718142acb","Type":"ContainerStarted","Data":"657dca19fb05d78b915e8c96a27fdbfdf137e39e00243475106046041fe25270"} Sep 10 13:41:59.644714 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:59.644708 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:41:59.675150 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:41:59.675110 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" podStartSLOduration=1.15440959 podStartE2EDuration="2.675096584s" totalImagesPullingTime="1.520686994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:59.674509695 +0000 UTC m=+797.743509659" watchObservedRunningTime="2026-09-10 13:41:59.675096584 +0000 UTC m=+797.744096547" Sep 10 13:42:10.652090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:10.652013 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mbq7f" Sep 10 13:42:28.907040 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.907009 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf"] Sep 10 13:42:28.910405 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.910387 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:28.914218 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.914195 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 13:42:28.914342 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.914195 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 13:42:28.914342 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.914199 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vzhxx\"" Sep 10 13:42:28.914433 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.914263 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 13:42:28.922267 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:28.922246 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf"] Sep 10 13:42:29.010933 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.010901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhzt\" (UniqueName: \"kubernetes.io/projected/579765f9-0be9-448b-90cc-3b4ccdb86ecd-kube-api-access-mfhzt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.011087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.010955 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.011087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.011035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.011087 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.011060 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.011187 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.011088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.011187 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.011108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579765f9-0be9-448b-90cc-3b4ccdb86ecd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.111752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.111718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.111752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.111757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579765f9-0be9-448b-90cc-3b4ccdb86ecd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.111984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.111791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhzt\" (UniqueName: \"kubernetes.io/projected/579765f9-0be9-448b-90cc-3b4ccdb86ecd-kube-api-access-mfhzt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.111984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.111831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.111984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.111863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.111984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.111878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.112230 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.112209 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.112285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.112239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.112285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.112259 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.113919 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.113898 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579765f9-0be9-448b-90cc-3b4ccdb86ecd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.114201 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.114184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579765f9-0be9-448b-90cc-3b4ccdb86ecd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.126653 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.126629 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhzt\" (UniqueName: \"kubernetes.io/projected/579765f9-0be9-448b-90cc-3b4ccdb86ecd-kube-api-access-mfhzt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sv6cf\" (UID: \"579765f9-0be9-448b-90cc-3b4ccdb86ecd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.220478 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.220450 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:29.509103 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.509078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf"] Sep 10 13:42:29.510214 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:42:29.510191 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579765f9_0be9_448b_90cc_3b4ccdb86ecd.slice/crio-b444dbe0eb014aa88db07d0b13b5fe92bc8f7c7f0d4dedb8821eda92ae06bfcf WatchSource:0}: Error finding container b444dbe0eb014aa88db07d0b13b5fe92bc8f7c7f0d4dedb8821eda92ae06bfcf: Status 404 returned error can't find the container with id b444dbe0eb014aa88db07d0b13b5fe92bc8f7c7f0d4dedb8821eda92ae06bfcf Sep 10 13:42:29.787506 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:29.787434 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" event={"ID":"579765f9-0be9-448b-90cc-3b4ccdb86ecd","Type":"ContainerStarted","Data":"b444dbe0eb014aa88db07d0b13b5fe92bc8f7c7f0d4dedb8821eda92ae06bfcf"} Sep 10 13:42:30.794982 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:30.794904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" event={"ID":"579765f9-0be9-448b-90cc-3b4ccdb86ecd","Type":"ContainerStarted","Data":"019ea096ba36b1fbca247a5cfdf2258842ab40183c4667d8b8fd3d09d3fdc26e"} Sep 10 13:42:30.857197 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:30.857149 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" podStartSLOduration=1.824881204 podStartE2EDuration="2.857135235s" totalImagesPullingTime="1.032254031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:42:30.855426855 +0000 UTC m=+828.924426818" watchObservedRunningTime="2026-09-10 13:42:30.857135235 +0000 UTC m=+828.926135218" Sep 10 13:42:39.221546 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:39.221513 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:39.221995 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:39.221614 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:39.233406 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:39.233381 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:39.849497 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:39.849472 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sv6cf" Sep 10 13:42:56.759527 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.759493 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4"] Sep 10 13:42:56.762956 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.762932 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:56.766657 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.766637 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 10 13:42:56.787204 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.787179 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4"] Sep 10 13:42:56.941110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.941080 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:56.941110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.941114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgj8m\" (UniqueName: \"kubernetes.io/projected/6e40bd3c-06f9-41f0-8385-52927b807735-kube-api-access-wgj8m\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:56.941313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.941185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:56.941313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.941205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:56.941313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.941251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:56.941313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:56.941289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e40bd3c-06f9-41f0-8385-52927b807735-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042518 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e40bd3c-06f9-41f0-8385-52927b807735-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042774 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042570 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042774 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgj8m\" (UniqueName: \"kubernetes.io/projected/6e40bd3c-06f9-41f0-8385-52927b807735-kube-api-access-wgj8m\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042876 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042853 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.042950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042932 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.043003 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.042979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.043185 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.043158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.044824 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.044804 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e40bd3c-06f9-41f0-8385-52927b807735-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.044986 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.044969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e40bd3c-06f9-41f0-8385-52927b807735-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.055393 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.055371 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgj8m\" (UniqueName: \"kubernetes.io/projected/6e40bd3c-06f9-41f0-8385-52927b807735-kube-api-access-wgj8m\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4\" (UID: \"6e40bd3c-06f9-41f0-8385-52927b807735\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.072185 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.072165 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:42:57.355011 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.354985 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4"] Sep 10 13:42:57.356650 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:42:57.356619 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e40bd3c_06f9_41f0_8385_52927b807735.slice/crio-f7fbee9474932c598fbec1b702907f0283ba4f799301490e4660b9e6bdb58423 WatchSource:0}: Error finding container f7fbee9474932c598fbec1b702907f0283ba4f799301490e4660b9e6bdb58423: Status 404 returned error can't find the container with id f7fbee9474932c598fbec1b702907f0283ba4f799301490e4660b9e6bdb58423 Sep 10 13:42:57.923839 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.923802 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" event={"ID":"6e40bd3c-06f9-41f0-8385-52927b807735","Type":"ContainerStarted","Data":"c217ca07e695971a209a2b06c890ba6c5f474248de889e96823a425a5ea33a95"} Sep 10 13:42:57.924210 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:42:57.923854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" event={"ID":"6e40bd3c-06f9-41f0-8385-52927b807735","Type":"ContainerStarted","Data":"f7fbee9474932c598fbec1b702907f0283ba4f799301490e4660b9e6bdb58423"} Sep 10 13:43:07.072896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:07.072847 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:43:07.072896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:07.072917 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:43:07.085783 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:07.085751 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:43:07.113532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:07.113481 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" podStartSLOduration=10.924216598 podStartE2EDuration="11.113465802s" totalImagesPullingTime="189.249204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:42:57.972952386 +0000 UTC m=+856.041952350" watchObservedRunningTime="2026-09-10 13:43:07.113465802 +0000 UTC m=+865.182465768" Sep 10 13:43:07.982361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:07.982329 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4" Sep 10 13:43:42.372404 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:42.372313 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:43:42.374855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:42.374826 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:43:42.381404 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:42.381375 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:43:42.383384 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:43:42.383364 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:45:00.153664 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.153631 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg"] Sep 10 13:45:00.156883 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.156867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" Sep 10 13:45:00.160069 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.160045 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:45:00.164300 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.164276 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg"] Sep 10 13:45:00.167309 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.167286 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" Sep 10 13:45:00.442071 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.442046 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg"] Sep 10 13:45:00.443306 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:45:00.443274 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8232f2c2_ee24_4972_81a8_7fa175c21f74.slice/crio-6ba577c4ebc4a20745fdea3a6d82e182e48006f656ed8dafadc4765868989178 WatchSource:0}: Error finding container 6ba577c4ebc4a20745fdea3a6d82e182e48006f656ed8dafadc4765868989178: Status 404 returned error can't find the container with id 6ba577c4ebc4a20745fdea3a6d82e182e48006f656ed8dafadc4765868989178 Sep 10 13:45:00.513348 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:00.513305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" event={"ID":"8232f2c2-ee24-4972-81a8-7fa175c21f74","Type":"ContainerStarted","Data":"6ba577c4ebc4a20745fdea3a6d82e182e48006f656ed8dafadc4765868989178"} Sep 10 13:45:03.530079 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:03.530043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" event={"ID":"8232f2c2-ee24-4972-81a8-7fa175c21f74","Type":"ContainerStarted","Data":"bb89d5b60436b368e1da8392e9d6372e3dbfc1a7c77e0a6d675f90157bf5d3a5"} Sep 10 13:45:04.536403 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:04.536367 2568 generic.go:356] "Generic (PLEG): container finished" podID="8232f2c2-ee24-4972-81a8-7fa175c21f74" containerID="bb89d5b60436b368e1da8392e9d6372e3dbfc1a7c77e0a6d675f90157bf5d3a5" exitCode=0 Sep 10 13:45:04.536865 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:04.536451 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" event={"ID":"8232f2c2-ee24-4972-81a8-7fa175c21f74","Type":"ContainerDied","Data":"bb89d5b60436b368e1da8392e9d6372e3dbfc1a7c77e0a6d675f90157bf5d3a5"} Sep 10 13:45:05.672232 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:05.672210 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" Sep 10 13:45:06.547231 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:06.547199 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" Sep 10 13:45:06.547407 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:06.547196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-pvlfg" event={"ID":"8232f2c2-ee24-4972-81a8-7fa175c21f74","Type":"ContainerDied","Data":"6ba577c4ebc4a20745fdea3a6d82e182e48006f656ed8dafadc4765868989178"} Sep 10 13:45:06.547407 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:06.547329 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ba577c4ebc4a20745fdea3a6d82e182e48006f656ed8dafadc4765868989178" Sep 10 13:45:33.149858 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.149826 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl"] Sep 10 13:45:33.150234 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.150192 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8232f2c2-ee24-4972-81a8-7fa175c21f74" containerName="cleanup" Sep 10 13:45:33.150234 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.150204 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8232f2c2-ee24-4972-81a8-7fa175c21f74" containerName="cleanup" Sep 10 13:45:33.150307 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.150266 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8232f2c2-ee24-4972-81a8-7fa175c21f74" containerName="cleanup" Sep 10 13:45:33.153356 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.153335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.155998 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.155981 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-86fdcf-openshift-default-dockercfg-4crwt\"" Sep 10 13:45:33.165252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.165232 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl"] Sep 10 13:45:33.286394 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-podinfo\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-envoy\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286471 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-certs\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-socket\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-credential-socket\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286543 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istiod-ca-cert\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286566 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-data\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286794 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286582 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-token\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.286794 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.286620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4h7\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-kube-api-access-2r4h7\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387231 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-credential-socket\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istiod-ca-cert\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-data\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-token\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4h7\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-kube-api-access-2r4h7\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-podinfo\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-envoy\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-certs\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-credential-socket\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387702 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-data\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.387912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-socket\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.388072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387909 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-certs\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.388072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.387969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-socket\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.388161 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.388137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istiod-ca-cert\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.389814 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.389788 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-envoy\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.390009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.389989 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-podinfo\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.422186 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.422117 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-token\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.422313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.422197 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4h7\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-kube-api-access-2r4h7\") pod \"e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.464655 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.464627 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:33.755541 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.755513 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl"] Sep 10 13:45:33.757219 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:45:33.757189 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f0ae13_c5d1_47bd_b492_821ad53e08e4.slice/crio-d25897f6b436c3af1a559840f5cfa71cf9e7ac99c36df4b85ed6513e138574ca WatchSource:0}: Error finding container d25897f6b436c3af1a559840f5cfa71cf9e7ac99c36df4b85ed6513e138574ca: Status 404 returned error can't find the container with id d25897f6b436c3af1a559840f5cfa71cf9e7ac99c36df4b85ed6513e138574ca Sep 10 13:45:33.759526 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.759494 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:33.759593 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.759563 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:33.759651 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:33.759618 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:34.691079 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:34.691043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" event={"ID":"56f0ae13-c5d1-47bd-b492-821ad53e08e4","Type":"ContainerStarted","Data":"451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752"} Sep 10 13:45:34.691548 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:34.691098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" event={"ID":"56f0ae13-c5d1-47bd-b492-821ad53e08e4","Type":"ContainerStarted","Data":"d25897f6b436c3af1a559840f5cfa71cf9e7ac99c36df4b85ed6513e138574ca"} Sep 10 13:45:34.738808 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:34.738760 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podStartSLOduration=1.738739128 podStartE2EDuration="1.738739128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:34.737285992 +0000 UTC m=+1012.806285956" watchObservedRunningTime="2026-09-10 13:45:34.738739128 +0000 UTC m=+1012.807739095" Sep 10 13:45:35.465678 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:35.465638 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:35.470538 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:35.470516 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:35.696534 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:35.696502 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:35.697346 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:35.697329 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:38.950307 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:38.950272 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2"] Sep 10 13:45:38.957838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:38.957814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:38.960371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:38.960348 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-86fdcf\"" Sep 10 13:45:38.960574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:38.960420 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-86fdcf-dockercfg-wlbfm\"" Sep 10 13:45:38.964557 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:38.964505 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2"] Sep 10 13:45:39.035578 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.035538 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de254e-4752-45df-81af-3336099b3d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.035766 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.035586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbrh\" (UniqueName: \"kubernetes.io/projected/43de254e-4752-45df-81af-3336099b3d44-kube-api-access-pfbrh\") pod \"payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.136850 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.136810 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de254e-4752-45df-81af-3336099b3d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.136850 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.136850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbrh\" (UniqueName: \"kubernetes.io/projected/43de254e-4752-45df-81af-3336099b3d44-kube-api-access-pfbrh\") pod \"payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.137664 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.137639 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de254e-4752-45df-81af-3336099b3d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.146254 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.146228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbrh\" (UniqueName: \"kubernetes.io/projected/43de254e-4752-45df-81af-3336099b3d44-kube-api-access-pfbrh\") pod \"payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.270453 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.270414 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.545776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.545745 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2"] Sep 10 13:45:39.547632 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:45:39.547584 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43de254e_4752_45df_81af_3336099b3d44.slice/crio-5a885409bb65342b47945e5c9e66887d039b51806e109fd08d85de2d2b21a3e2 WatchSource:0}: Error finding container 5a885409bb65342b47945e5c9e66887d039b51806e109fd08d85de2d2b21a3e2: Status 404 returned error can't find the container with id 5a885409bb65342b47945e5c9e66887d039b51806e109fd08d85de2d2b21a3e2 Sep 10 13:45:39.720878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.720846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" event={"ID":"43de254e-4752-45df-81af-3336099b3d44","Type":"ContainerStarted","Data":"88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb"} Sep 10 13:45:39.721035 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.720896 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" event={"ID":"43de254e-4752-45df-81af-3336099b3d44","Type":"ContainerStarted","Data":"5a885409bb65342b47945e5c9e66887d039b51806e109fd08d85de2d2b21a3e2"} Sep 10 13:45:39.721035 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.720969 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:39.747082 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:39.747029 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" podStartSLOduration=1.7470147169999999 podStartE2EDuration="1.747014717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:39.746720091 +0000 UTC m=+1017.815720055" watchObservedRunningTime="2026-09-10 13:45:39.747014717 +0000 UTC m=+1017.816014678" Sep 10 13:45:41.896946 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.896910 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w"] Sep 10 13:45:41.900488 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.900472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.903838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.903815 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b1f490-openshift-default-dockercfg-vhdtn\"" Sep 10 13:45:41.916767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.916746 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w"] Sep 10 13:45:41.964426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964374 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-socket\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-certs\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-envoy\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-podinfo\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964520 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-data\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istiod-ca-cert\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-token\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmvr\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-kube-api-access-8pmvr\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:41.964965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:41.964769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-credential-socket\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-token\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmvr\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-kube-api-access-8pmvr\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065501 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-credential-socket\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-socket\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-certs\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-envoy\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-podinfo\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-data\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.065725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.065702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istiod-ca-cert\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.066110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.066084 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-socket\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.066177 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.066097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-certs\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.066285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.066268 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-credential-socket\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.066357 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.066310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-data\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.066506 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.066486 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istiod-ca-cert\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.067923 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.067901 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-envoy\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.068428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.068407 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-podinfo\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.080669 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.080644 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-token\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.082631 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.082607 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmvr\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-kube-api-access-8pmvr\") pod \"e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.210809 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.210723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:42.497444 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:45:42.497413 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ef1faa_d5f2_4690_9cc3_812dcbef8001.slice/crio-55aaaaa8dcdde63b1c7ba7ab744a14ddea1792bbb7c121fae23c6f81f56a7e9d WatchSource:0}: Error finding container 55aaaaa8dcdde63b1c7ba7ab744a14ddea1792bbb7c121fae23c6f81f56a7e9d: Status 404 returned error can't find the container with id 55aaaaa8dcdde63b1c7ba7ab744a14ddea1792bbb7c121fae23c6f81f56a7e9d Sep 10 13:45:42.499616 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.499536 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:42.499700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.499653 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:42.499700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.499686 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:42.503705 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.503679 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w"] Sep 10 13:45:42.749627 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.749511 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" event={"ID":"e2ef1faa-d5f2-4690-9cc3-812dcbef8001","Type":"ContainerStarted","Data":"4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286"} Sep 10 13:45:42.750049 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.749641 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" event={"ID":"e2ef1faa-d5f2-4690-9cc3-812dcbef8001","Type":"ContainerStarted","Data":"55aaaaa8dcdde63b1c7ba7ab744a14ddea1792bbb7c121fae23c6f81f56a7e9d"} Sep 10 13:45:42.777992 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:42.777940 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podStartSLOduration=1.77792503 podStartE2EDuration="1.77792503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:42.774367897 +0000 UTC m=+1020.843367861" watchObservedRunningTime="2026-09-10 13:45:42.77792503 +0000 UTC m=+1020.846924995" Sep 10 13:45:43.211490 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:43.211405 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:43.216195 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:43.216169 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:43.754905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:43.754874 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:43.755824 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:43.755805 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:45:49.895773 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.895742 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z"] Sep 10 13:45:49.899884 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.899859 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:49.909097 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.909068 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-854ac94d\"" Sep 10 13:45:49.926199 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.926174 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z"] Sep 10 13:45:49.941158 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.941119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jm6t\" (UniqueName: \"kubernetes.io/projected/06f2750d-302e-4d7d-a065-9aa53538cfdb-kube-api-access-5jm6t\") pod \"payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:49.941430 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.941404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f2750d-302e-4d7d-a065-9aa53538cfdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:49.981941 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.981909 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp"] Sep 10 13:45:49.985304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.985285 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:49.990759 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:49.990741 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-854ac94d-dockercfg-68c5l\"" Sep 10 13:45:50.000725 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.000698 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp"] Sep 10 13:45:50.046096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.046053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jm6t\" (UniqueName: \"kubernetes.io/projected/06f2750d-302e-4d7d-a065-9aa53538cfdb-kube-api-access-5jm6t\") pod \"payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:50.046372 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.046348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhmm\" (UniqueName: \"kubernetes.io/projected/ef3b9e7f-71de-410a-836a-01184a5074e2-kube-api-access-hfhmm\") pod \"payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.046532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.046514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f2750d-302e-4d7d-a065-9aa53538cfdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:50.046747 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.046727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3b9e7f-71de-410a-836a-01184a5074e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.048141 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.048117 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f2750d-302e-4d7d-a065-9aa53538cfdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:50.073836 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.073799 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jm6t\" (UniqueName: \"kubernetes.io/projected/06f2750d-302e-4d7d-a065-9aa53538cfdb-kube-api-access-5jm6t\") pod \"payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:50.148060 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.147964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3b9e7f-71de-410a-836a-01184a5074e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.148369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.148348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhmm\" (UniqueName: \"kubernetes.io/projected/ef3b9e7f-71de-410a-836a-01184a5074e2-kube-api-access-hfhmm\") pod \"payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.149628 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.149418 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3b9e7f-71de-410a-836a-01184a5074e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.162002 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.161977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhmm\" (UniqueName: \"kubernetes.io/projected/ef3b9e7f-71de-410a-836a-01184a5074e2-kube-api-access-hfhmm\") pod \"payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.212491 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.212463 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:50.294991 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.294954 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.514477 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.514453 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z"] Sep 10 13:45:50.516136 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:45:50.516110 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f2750d_302e_4d7d_a065_9aa53538cfdb.slice/crio-55b40a2da2445ec86d5a8427ed548cf506c0faab937cfaca59fce9e465a3cc56 WatchSource:0}: Error finding container 55b40a2da2445ec86d5a8427ed548cf506c0faab937cfaca59fce9e465a3cc56: Status 404 returned error can't find the container with id 55b40a2da2445ec86d5a8427ed548cf506c0faab937cfaca59fce9e465a3cc56 Sep 10 13:45:50.627504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.627478 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp"] Sep 10 13:45:50.628780 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:45:50.628749 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3b9e7f_71de_410a_836a_01184a5074e2.slice/crio-b3c725a7000ed45652ea95d36c33965ac72447433ea000d6329cd524c247b856 WatchSource:0}: Error finding container b3c725a7000ed45652ea95d36c33965ac72447433ea000d6329cd524c247b856: Status 404 returned error can't find the container with id b3c725a7000ed45652ea95d36c33965ac72447433ea000d6329cd524c247b856 Sep 10 13:45:50.728322 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.728295 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:45:50.795050 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.794962 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" event={"ID":"ef3b9e7f-71de-410a-836a-01184a5074e2","Type":"ContainerStarted","Data":"e861bd7489fe3fe7a3055b62f2dc16e27628b356c2c12200bc123021ca8dd97f"} Sep 10 13:45:50.795346 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.795326 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:45:50.795499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.795482 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" event={"ID":"ef3b9e7f-71de-410a-836a-01184a5074e2","Type":"ContainerStarted","Data":"b3c725a7000ed45652ea95d36c33965ac72447433ea000d6329cd524c247b856"} Sep 10 13:45:50.797662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.797143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" event={"ID":"06f2750d-302e-4d7d-a065-9aa53538cfdb","Type":"ContainerStarted","Data":"c8d902ca28327027d2e41f3d2165cc60b7a27fb234b371818ba90a0a212177d3"} Sep 10 13:45:50.797662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.797190 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" event={"ID":"06f2750d-302e-4d7d-a065-9aa53538cfdb","Type":"ContainerStarted","Data":"55b40a2da2445ec86d5a8427ed548cf506c0faab937cfaca59fce9e465a3cc56"} Sep 10 13:45:50.797901 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.797725 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:45:50.913724 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.913672 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" podStartSLOduration=1.913654464 podStartE2EDuration="1.913654464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:50.9110473 +0000 UTC m=+1028.980047265" watchObservedRunningTime="2026-09-10 13:45:50.913654464 +0000 UTC m=+1028.982654426" Sep 10 13:45:50.963053 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:50.962989 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" podStartSLOduration=1.962967388 podStartE2EDuration="1.962967388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:50.950175379 +0000 UTC m=+1029.019175345" watchObservedRunningTime="2026-09-10 13:45:50.962967388 +0000 UTC m=+1029.031967355" Sep 10 13:45:54.821443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:54.821407 2568 generic.go:356] "Generic (PLEG): container finished" podID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerID="451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752" exitCode=0 Sep 10 13:45:54.821871 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:54.821485 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" event={"ID":"56f0ae13-c5d1-47bd-b492-821ad53e08e4","Type":"ContainerDied","Data":"451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752"} Sep 10 13:45:54.821963 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:54.821932 2568 scope.go:117] "RemoveContainer" containerID="451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752" Sep 10 13:45:54.823117 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:54.823083 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:54.823212 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:54.823173 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:54.823274 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:54.823229 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:45:55.828511 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:55.828483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" event={"ID":"56f0ae13-c5d1-47bd-b492-821ad53e08e4","Type":"ContainerStarted","Data":"e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d"} Sep 10 13:45:56.466545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:56.466098 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:45:56.466545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:56.466229 2568 patch_prober.go:28] interesting pod/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 10 13:45:56.466545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:56.466275 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 10 13:45:57.466178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:57.465700 2568 patch_prober.go:28] interesting pod/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 10 13:45:57.466178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:57.465772 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 10 13:45:58.465128 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:58.465085 2568 patch_prober.go:28] interesting pod/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 10 13:45:58.465324 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:58.465156 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 10 13:45:59.465819 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:59.465773 2568 patch_prober.go:28] interesting pod/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 10 13:45:59.466230 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:59.465836 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 10 13:45:59.795883 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:59.795787 2568 patch_prober.go:28] interesting pod/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:45:59.795883 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:45:59.795868 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:46:00.465535 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:00.465503 2568 patch_prober.go:28] interesting pod/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 10 13:46:00.465799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:00.465778 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 10 13:46:01.469172 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:01.469142 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:46:01.469635 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:01.469433 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:46:01.470115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:01.470099 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:46:02.806863 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.806820 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6"] Sep 10 13:46:02.810234 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.810211 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.811281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.811247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:46:02.811536 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.811516 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:46:02.814077 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.814059 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-02fc2e-openshift-default-dockercfg-66jtt\"" Sep 10 13:46:02.828883 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.828858 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6"] Sep 10 13:46:02.885195 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.885158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-credential-socket\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.885853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.885817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-podinfo\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.885904 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-socket\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.885989 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-token\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886156 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.886136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-envoy\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.886192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-data\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.886226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770d524e-88c0-4e57-95b6-a9d596155ca2-istiod-ca-cert\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.886270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-certs\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.886522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.886357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2754\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-kube-api-access-d2754\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.892821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.892774 2568 generic.go:356] "Generic (PLEG): container finished" podID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerID="4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286" exitCode=0 Sep 10 13:46:02.893042 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.893014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" event={"ID":"e2ef1faa-d5f2-4690-9cc3-812dcbef8001","Type":"ContainerDied","Data":"4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286"} Sep 10 13:46:02.894120 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.894100 2568 scope.go:117] "RemoveContainer" containerID="4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286" Sep 10 13:46:02.897519 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.897279 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:02.897519 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.897356 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:02.897519 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.897397 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.987769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2754\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-kube-api-access-d2754\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.987838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-credential-socket\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.987866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-podinfo\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.987908 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-socket\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.987948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-token\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-envoy\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-data\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770d524e-88c0-4e57-95b6-a9d596155ca2-istiod-ca-cert\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-certs\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-credential-socket\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988782 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-data\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.988862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.988835 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-certs\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.989167 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.989144 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-socket\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.989727 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.989701 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770d524e-88c0-4e57-95b6-a9d596155ca2-istiod-ca-cert\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.991515 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.991484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-envoy\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:02.991962 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:02.991940 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-podinfo\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:03.003956 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.003934 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-token\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:03.010305 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.010277 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2754\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-kube-api-access-d2754\") pod \"e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:03.121818 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.121740 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:03.436994 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.436933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6"] Sep 10 13:46:03.439548 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.439514 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:03.439662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.439592 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:03.439738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.439700 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:03.900078 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.900037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" event={"ID":"e2ef1faa-d5f2-4690-9cc3-812dcbef8001","Type":"ContainerStarted","Data":"d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1"} Sep 10 13:46:03.901712 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.901684 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" event={"ID":"770d524e-88c0-4e57-95b6-a9d596155ca2","Type":"ContainerStarted","Data":"21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78"} Sep 10 13:46:03.901822 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:03.901725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" event={"ID":"770d524e-88c0-4e57-95b6-a9d596155ca2","Type":"ContainerStarted","Data":"be6bd7c120b1e2697dc26d92cd56237ba631a0921832b5abaf44ea5d9b1b115d"} Sep 10 13:46:04.003104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.003048 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podStartSLOduration=2.003031815 podStartE2EDuration="2.003031815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:04.000546087 +0000 UTC m=+1042.069546073" watchObservedRunningTime="2026-09-10 13:46:04.003031815 +0000 UTC m=+1042.072031783" Sep 10 13:46:04.122664 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.122624 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:04.128019 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.127991 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:04.211283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.211199 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:46:04.211427 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.211360 2568 patch_prober.go:28] interesting pod/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 13:46:04.211469 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.211420 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 13:46:04.907452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.907416 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:04.908317 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:04.908298 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:05.211776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:05.211686 2568 patch_prober.go:28] interesting pod/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 13:46:05.211776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:05.211747 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 13:46:06.211560 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:06.211518 2568 patch_prober.go:28] interesting pod/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 13:46:06.211991 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:06.211585 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 13:46:07.211976 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:07.211934 2568 patch_prober.go:28] interesting pod/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 13:46:07.212377 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:07.211995 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 13:46:08.211443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:08.211404 2568 patch_prober.go:28] interesting pod/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 13:46:08.211677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:08.211469 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 13:46:09.309570 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:09.309524 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:46:09.310088 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:09.309968 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:46:09.311335 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:09.311309 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:46:17.588621 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.588556 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s"] Sep 10 13:46:17.592611 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.592579 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:17.597445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.597420 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:46:17.597708 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.597689 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:46:17.609771 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.609740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s"] Sep 10 13:46:17.654418 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.654372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39128dad-7da8-4ea2-b525-8f3dcb9c7242-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ac94d-cz59s\" (UID: \"39128dad-7da8-4ea2-b525-8f3dcb9c7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:17.755646 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.755510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39128dad-7da8-4ea2-b525-8f3dcb9c7242-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ac94d-cz59s\" (UID: \"39128dad-7da8-4ea2-b525-8f3dcb9c7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:17.756264 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.756238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39128dad-7da8-4ea2-b525-8f3dcb9c7242-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ac94d-cz59s\" (UID: \"39128dad-7da8-4ea2-b525-8f3dcb9c7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:17.903375 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:17.903282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:18.206080 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:18.206051 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s"] Sep 10 13:46:18.207666 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:46:18.207631 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39128dad_7da8_4ea2_b525_8f3dcb9c7242.slice/crio-e84cb740db19228ef326437b6bc421a2e4028aa837fcf88e9ae91cdb91e92e0e WatchSource:0}: Error finding container e84cb740db19228ef326437b6bc421a2e4028aa837fcf88e9ae91cdb91e92e0e: Status 404 returned error can't find the container with id e84cb740db19228ef326437b6bc421a2e4028aa837fcf88e9ae91cdb91e92e0e Sep 10 13:46:18.993425 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:18.993378 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" event={"ID":"39128dad-7da8-4ea2-b525-8f3dcb9c7242","Type":"ContainerStarted","Data":"97108d68bf4dbf458e6349a0c6db318860d7e8a487e069f19db74f5ef9969df2"} Sep 10 13:46:18.993867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:18.993483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" event={"ID":"39128dad-7da8-4ea2-b525-8f3dcb9c7242","Type":"ContainerStarted","Data":"e84cb740db19228ef326437b6bc421a2e4028aa837fcf88e9ae91cdb91e92e0e"} Sep 10 13:46:19.014481 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:19.014419 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" podStartSLOduration=2.014400405 podStartE2EDuration="2.014400405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:19.012472644 +0000 UTC m=+1057.081472608" watchObservedRunningTime="2026-09-10 13:46:19.014400405 +0000 UTC m=+1057.083400369" Sep 10 13:46:20.000044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:20.000009 2568 generic.go:356] "Generic (PLEG): container finished" podID="39128dad-7da8-4ea2-b525-8f3dcb9c7242" containerID="97108d68bf4dbf458e6349a0c6db318860d7e8a487e069f19db74f5ef9969df2" exitCode=0 Sep 10 13:46:20.000442 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:20.000099 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" event={"ID":"39128dad-7da8-4ea2-b525-8f3dcb9c7242","Type":"ContainerDied","Data":"97108d68bf4dbf458e6349a0c6db318860d7e8a487e069f19db74f5ef9969df2"} Sep 10 13:46:21.185835 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:21.185809 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:21.286359 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:21.286271 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39128dad-7da8-4ea2-b525-8f3dcb9c7242-maas-api-service-ca\") pod \"39128dad-7da8-4ea2-b525-8f3dcb9c7242\" (UID: \"39128dad-7da8-4ea2-b525-8f3dcb9c7242\") " Sep 10 13:46:21.286609 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:21.286574 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39128dad-7da8-4ea2-b525-8f3dcb9c7242-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39128dad-7da8-4ea2-b525-8f3dcb9c7242" (UID: "39128dad-7da8-4ea2-b525-8f3dcb9c7242"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:21.387323 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:21.387277 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39128dad-7da8-4ea2-b525-8f3dcb9c7242-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:46:22.013148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:22.013118 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" Sep 10 13:46:22.013338 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:22.013108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s" event={"ID":"39128dad-7da8-4ea2-b525-8f3dcb9c7242","Type":"ContainerDied","Data":"e84cb740db19228ef326437b6bc421a2e4028aa837fcf88e9ae91cdb91e92e0e"} Sep 10 13:46:22.013338 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:22.013256 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84cb740db19228ef326437b6bc421a2e4028aa837fcf88e9ae91cdb91e92e0e" Sep 10 13:46:25.037892 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:25.037855 2568 generic.go:356] "Generic (PLEG): container finished" podID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerID="21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78" exitCode=0 Sep 10 13:46:25.038378 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:25.037935 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" event={"ID":"770d524e-88c0-4e57-95b6-a9d596155ca2","Type":"ContainerDied","Data":"21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78"} Sep 10 13:46:25.038490 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:25.038471 2568 scope.go:117] "RemoveContainer" containerID="21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78" Sep 10 13:46:25.039788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:25.039750 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:25.039885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:25.039831 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:25.039885 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:25.039871 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:26.049026 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:26.048989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" event={"ID":"770d524e-88c0-4e57-95b6-a9d596155ca2","Type":"ContainerStarted","Data":"351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f"} Sep 10 13:46:26.122701 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:26.122656 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:26.123004 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:26.122979 2568 patch_prober.go:28] interesting pod/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 10 13:46:26.123053 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:26.123032 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 10 13:46:27.122784 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:27.122749 2568 patch_prober.go:28] interesting pod/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 10 13:46:27.123199 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:27.122811 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 10 13:46:28.122704 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:28.122663 2568 patch_prober.go:28] interesting pod/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 10 13:46:28.122914 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:28.122721 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 10 13:46:28.886194 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:28.886160 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp"] Sep 10 13:46:28.886463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:28.886429 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" podUID="ef3b9e7f-71de-410a-836a-01184a5074e2" containerName="payload-processing" containerID="cri-o://e861bd7489fe3fe7a3055b62f2dc16e27628b356c2c12200bc123021ca8dd97f" gracePeriod=30 Sep 10 13:46:28.892958 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:28.892930 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z"] Sep 10 13:46:28.893366 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:28.893338 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" podUID="06f2750d-302e-4d7d-a065-9aa53538cfdb" containerName="payload-pre-processing" containerID="cri-o://c8d902ca28327027d2e41f3d2165cc60b7a27fb234b371818ba90a0a212177d3" gracePeriod=30 Sep 10 13:46:29.070258 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.070220 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f2750d-302e-4d7d-a065-9aa53538cfdb" containerID="c8d902ca28327027d2e41f3d2165cc60b7a27fb234b371818ba90a0a212177d3" exitCode=0 Sep 10 13:46:29.070461 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.070298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" event={"ID":"06f2750d-302e-4d7d-a065-9aa53538cfdb","Type":"ContainerDied","Data":"c8d902ca28327027d2e41f3d2165cc60b7a27fb234b371818ba90a0a212177d3"} Sep 10 13:46:29.072122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.072098 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef3b9e7f-71de-410a-836a-01184a5074e2" containerID="e861bd7489fe3fe7a3055b62f2dc16e27628b356c2c12200bc123021ca8dd97f" exitCode=0 Sep 10 13:46:29.072252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.072174 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" event={"ID":"ef3b9e7f-71de-410a-836a-01184a5074e2","Type":"ContainerDied","Data":"e861bd7489fe3fe7a3055b62f2dc16e27628b356c2c12200bc123021ca8dd97f"} Sep 10 13:46:29.122331 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.122295 2568 patch_prober.go:28] interesting pod/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 10 13:46:29.122506 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.122363 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 10 13:46:29.197133 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.197102 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:46:29.217549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.217512 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s"] Sep 10 13:46:29.225931 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.225901 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ac94d-cz59s"] Sep 10 13:46:29.264441 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.264413 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:46:29.272453 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.272425 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3b9e7f-71de-410a-836a-01184a5074e2-plugins-config-volume\") pod \"ef3b9e7f-71de-410a-836a-01184a5074e2\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " Sep 10 13:46:29.272658 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.272518 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfhmm\" (UniqueName: \"kubernetes.io/projected/ef3b9e7f-71de-410a-836a-01184a5074e2-kube-api-access-hfhmm\") pod \"ef3b9e7f-71de-410a-836a-01184a5074e2\" (UID: \"ef3b9e7f-71de-410a-836a-01184a5074e2\") " Sep 10 13:46:29.272935 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.272905 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3b9e7f-71de-410a-836a-01184a5074e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef3b9e7f-71de-410a-836a-01184a5074e2" (UID: "ef3b9e7f-71de-410a-836a-01184a5074e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:29.274970 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.274944 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3b9e7f-71de-410a-836a-01184a5074e2-kube-api-access-hfhmm" (OuterVolumeSpecName: "kube-api-access-hfhmm") pod "ef3b9e7f-71de-410a-836a-01184a5074e2" (UID: "ef3b9e7f-71de-410a-836a-01184a5074e2"). InnerVolumeSpecName "kube-api-access-hfhmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:29.374105 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.374069 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jm6t\" (UniqueName: \"kubernetes.io/projected/06f2750d-302e-4d7d-a065-9aa53538cfdb-kube-api-access-5jm6t\") pod \"06f2750d-302e-4d7d-a065-9aa53538cfdb\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " Sep 10 13:46:29.374306 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.374120 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f2750d-302e-4d7d-a065-9aa53538cfdb-plugins-config-volume\") pod \"06f2750d-302e-4d7d-a065-9aa53538cfdb\" (UID: \"06f2750d-302e-4d7d-a065-9aa53538cfdb\") " Sep 10 13:46:29.374384 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.374354 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3b9e7f-71de-410a-836a-01184a5074e2-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:46:29.374384 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.374367 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhmm\" (UniqueName: \"kubernetes.io/projected/ef3b9e7f-71de-410a-836a-01184a5074e2-kube-api-access-hfhmm\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:46:29.374704 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.374679 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f2750d-302e-4d7d-a065-9aa53538cfdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06f2750d-302e-4d7d-a065-9aa53538cfdb" (UID: "06f2750d-302e-4d7d-a065-9aa53538cfdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:29.376911 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.376880 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f2750d-302e-4d7d-a065-9aa53538cfdb-kube-api-access-5jm6t" (OuterVolumeSpecName: "kube-api-access-5jm6t") pod "06f2750d-302e-4d7d-a065-9aa53538cfdb" (UID: "06f2750d-302e-4d7d-a065-9aa53538cfdb"). InnerVolumeSpecName "kube-api-access-5jm6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:29.475411 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.475365 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f2750d-302e-4d7d-a065-9aa53538cfdb-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:46:29.475411 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:29.475412 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jm6t\" (UniqueName: \"kubernetes.io/projected/06f2750d-302e-4d7d-a065-9aa53538cfdb-kube-api-access-5jm6t\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:46:30.086891 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.086854 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" Sep 10 13:46:30.086891 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.086863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z" event={"ID":"06f2750d-302e-4d7d-a065-9aa53538cfdb","Type":"ContainerDied","Data":"55b40a2da2445ec86d5a8427ed548cf506c0faab937cfaca59fce9e465a3cc56"} Sep 10 13:46:30.087105 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.086965 2568 scope.go:117] "RemoveContainer" containerID="c8d902ca28327027d2e41f3d2165cc60b7a27fb234b371818ba90a0a212177d3" Sep 10 13:46:30.088395 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.088343 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" event={"ID":"ef3b9e7f-71de-410a-836a-01184a5074e2","Type":"ContainerDied","Data":"b3c725a7000ed45652ea95d36c33965ac72447433ea000d6329cd524c247b856"} Sep 10 13:46:30.088395 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.088369 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp" Sep 10 13:46:30.096850 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.096726 2568 scope.go:117] "RemoveContainer" containerID="e861bd7489fe3fe7a3055b62f2dc16e27628b356c2c12200bc123021ca8dd97f" Sep 10 13:46:30.115291 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.115263 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z"] Sep 10 13:46:30.121450 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.121421 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ac94d-5844ffd66b-qhj5z"] Sep 10 13:46:30.122390 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.122365 2568 patch_prober.go:28] interesting pod/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 10 13:46:30.122509 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.122412 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 10 13:46:30.150811 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.150774 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp"] Sep 10 13:46:30.155445 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.155330 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ac94d-85597c476c-vjfqp"] Sep 10 13:46:30.424658 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.424549 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f2750d-302e-4d7d-a065-9aa53538cfdb" path="/var/lib/kubelet/pods/06f2750d-302e-4d7d-a065-9aa53538cfdb/volumes" Sep 10 13:46:30.425010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.424958 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39128dad-7da8-4ea2-b525-8f3dcb9c7242" path="/var/lib/kubelet/pods/39128dad-7da8-4ea2-b525-8f3dcb9c7242/volumes" Sep 10 13:46:30.425281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.425267 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3b9e7f-71de-410a-836a-01184a5074e2" path="/var/lib/kubelet/pods/ef3b9e7f-71de-410a-836a-01184a5074e2/volumes" Sep 10 13:46:30.468801 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.468623 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5"] Sep 10 13:46:30.469200 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469183 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f2750d-302e-4d7d-a065-9aa53538cfdb" containerName="payload-pre-processing" Sep 10 13:46:30.469312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469202 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f2750d-302e-4d7d-a065-9aa53538cfdb" containerName="payload-pre-processing" Sep 10 13:46:30.469312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469216 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39128dad-7da8-4ea2-b525-8f3dcb9c7242" containerName="revoke-keys" Sep 10 13:46:30.469312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469226 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="39128dad-7da8-4ea2-b525-8f3dcb9c7242" containerName="revoke-keys" Sep 10 13:46:30.469312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469244 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3b9e7f-71de-410a-836a-01184a5074e2" containerName="payload-processing" Sep 10 13:46:30.469312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469253 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3b9e7f-71de-410a-836a-01184a5074e2" containerName="payload-processing" Sep 10 13:46:30.469590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469355 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3b9e7f-71de-410a-836a-01184a5074e2" containerName="payload-processing" Sep 10 13:46:30.469590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469371 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f2750d-302e-4d7d-a065-9aa53538cfdb" containerName="payload-pre-processing" Sep 10 13:46:30.469590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.469381 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="39128dad-7da8-4ea2-b525-8f3dcb9c7242" containerName="revoke-keys" Sep 10 13:46:30.474338 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.474313 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.478418 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.478379 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0e325272-gw-openshift-default-dockercfg-zptzn\"" Sep 10 13:46:30.488009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.487981 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5"] Sep 10 13:46:30.590290 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590248 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-data\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-socket\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bxn\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-kube-api-access-c7bxn\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-istio-token\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590430 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c7b888-5bb8-48be-85b5-e84922f87524-istiod-ca-cert\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590740 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-certs\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590740 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c7b888-5bb8-48be-85b5-e84922f87524-istio-podinfo\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590740 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590559 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-envoy\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.590740 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.590586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-credential-socket\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.691549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.691447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-certs\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.691549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.691484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c7b888-5bb8-48be-85b5-e84922f87524-istio-podinfo\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.691549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.691519 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-envoy\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.691873 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.691761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-credential-socket\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.691982 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.691964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-data\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692047 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-socket\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692070 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-credential-socket\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692118 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bxn\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-kube-api-access-c7bxn\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692219 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692128 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-istio-token\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692219 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c7b888-5bb8-48be-85b5-e84922f87524-istiod-ca-cert\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692337 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-data\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692355 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-socket\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692783 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692752 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-certs\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.692923 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.692902 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c7b888-5bb8-48be-85b5-e84922f87524-istiod-ca-cert\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.694261 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.694242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c7b888-5bb8-48be-85b5-e84922f87524-istio-podinfo\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.694343 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.694261 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-envoy\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.701973 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.701946 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-istio-token\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.703724 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.703704 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bxn\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-kube-api-access-c7bxn\") pod \"e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:30.787111 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:30.787072 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:31.096391 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.096347 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5"] Sep 10 13:46:31.098047 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:46:31.098005 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c7b888_5bb8_48be_85b5_e84922f87524.slice/crio-ad1c776e35aad4a8de75f26fbef1bd1384b35616ebdb6a68804413b3c4f8cc8b WatchSource:0}: Error finding container ad1c776e35aad4a8de75f26fbef1bd1384b35616ebdb6a68804413b3c4f8cc8b: Status 404 returned error can't find the container with id ad1c776e35aad4a8de75f26fbef1bd1384b35616ebdb6a68804413b3c4f8cc8b Sep 10 13:46:31.101235 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.101192 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:31.101343 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.101281 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:31.101343 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.101327 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:31.147138 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.147102 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:31.147930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.147860 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:31.149545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:31.149520 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:46:32.106513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:32.106477 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" event={"ID":"42c7b888-5bb8-48be-85b5-e84922f87524","Type":"ContainerStarted","Data":"f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61"} Sep 10 13:46:32.106959 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:32.106542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" event={"ID":"42c7b888-5bb8-48be-85b5-e84922f87524","Type":"ContainerStarted","Data":"ad1c776e35aad4a8de75f26fbef1bd1384b35616ebdb6a68804413b3c4f8cc8b"} Sep 10 13:46:32.139385 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:32.139332 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podStartSLOduration=2.139313713 podStartE2EDuration="2.139313713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:32.136086021 +0000 UTC m=+1070.205085986" watchObservedRunningTime="2026-09-10 13:46:32.139313713 +0000 UTC m=+1070.208313728" Sep 10 13:46:32.788025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:32.787983 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:32.793641 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:32.793608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:33.113277 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:33.113175 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:33.114352 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:33.114333 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:34.709273 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.709229 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb"] Sep 10 13:46:34.714892 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.714864 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:34.718006 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.717980 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0e325272\"" Sep 10 13:46:34.718139 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.717981 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0e325272-dockercfg-pk79c\"" Sep 10 13:46:34.724816 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.724786 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb"] Sep 10 13:46:34.831869 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.831778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bw72\" (UniqueName: \"kubernetes.io/projected/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-kube-api-access-8bw72\") pod \"payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:34.831869 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.831837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:34.932936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.932887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bw72\" (UniqueName: \"kubernetes.io/projected/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-kube-api-access-8bw72\") pod \"payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:34.933159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.932963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:34.933697 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.933676 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:34.943104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:34.943077 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bw72\" (UniqueName: \"kubernetes.io/projected/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-kube-api-access-8bw72\") pod \"payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:35.026984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:35.026948 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:35.309027 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:35.309000 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb"] Sep 10 13:46:35.311685 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:46:35.311649 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe9dd9b_213b_4f49_89ee_3fe805a3d245.slice/crio-6fc06189a2b158e9216524bc0b331a8e8b01bf3055e3b1aa4d0a962af79912ce WatchSource:0}: Error finding container 6fc06189a2b158e9216524bc0b331a8e8b01bf3055e3b1aa4d0a962af79912ce: Status 404 returned error can't find the container with id 6fc06189a2b158e9216524bc0b331a8e8b01bf3055e3b1aa4d0a962af79912ce Sep 10 13:46:36.131979 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:36.131940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" event={"ID":"6fe9dd9b-213b-4f49-89ee-3fe805a3d245","Type":"ContainerStarted","Data":"b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff"} Sep 10 13:46:36.132383 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:36.131998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" event={"ID":"6fe9dd9b-213b-4f49-89ee-3fe805a3d245","Type":"ContainerStarted","Data":"6fc06189a2b158e9216524bc0b331a8e8b01bf3055e3b1aa4d0a962af79912ce"} Sep 10 13:46:36.132383 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:36.132023 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:36.156653 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:36.156585 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" podStartSLOduration=2.156565803 podStartE2EDuration="2.156565803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:36.155863239 +0000 UTC m=+1074.224863209" watchObservedRunningTime="2026-09-10 13:46:36.156565803 +0000 UTC m=+1074.225565767" Sep 10 13:46:47.140383 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:47.140342 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:46:51.217636 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:51.217585 2568 generic.go:356] "Generic (PLEG): container finished" podID="42c7b888-5bb8-48be-85b5-e84922f87524" containerID="f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61" exitCode=0 Sep 10 13:46:51.218161 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:51.217664 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" event={"ID":"42c7b888-5bb8-48be-85b5-e84922f87524","Type":"ContainerDied","Data":"f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61"} Sep 10 13:46:51.218241 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:51.218220 2568 scope.go:117] "RemoveContainer" containerID="f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61" Sep 10 13:46:51.219401 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:51.219361 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:51.219497 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:51.219446 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:51.219536 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:51.219503 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:46:52.225001 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:52.224966 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" event={"ID":"42c7b888-5bb8-48be-85b5-e84922f87524","Type":"ContainerStarted","Data":"5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b"} Sep 10 13:46:52.787639 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:52.787585 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:52.787875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:52.787853 2568 patch_prober.go:28] interesting pod/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:46:52.787939 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:52.787906 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 13:46:53.788415 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:53.788373 2568 patch_prober.go:28] interesting pod/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:46:53.788844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:53.788452 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 13:46:54.787922 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:54.787885 2568 patch_prober.go:28] interesting pod/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:46:54.788116 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:54.787946 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 13:46:55.787838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:55.787802 2568 patch_prober.go:28] interesting pod/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:46:55.788242 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:55.787876 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 13:46:56.788046 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:56.788007 2568 patch_prober.go:28] interesting pod/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:46:56.788444 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:56.788073 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 13:46:57.791748 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:57.791709 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:57.792182 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:57.792065 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:57.792810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:57.792791 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:46:58.755369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.755328 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh"] Sep 10 13:46:58.759194 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.759172 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:46:58.762547 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.762525 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:46:58.762659 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.762646 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:46:58.770549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.770524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh"] Sep 10 13:46:58.893863 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.893822 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d598a-2562-48b2-a073-a37ba30d0d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh\" (UID: \"ad5d598a-2562-48b2-a073-a37ba30d0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:46:58.994960 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.994902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d598a-2562-48b2-a073-a37ba30d0d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh\" (UID: \"ad5d598a-2562-48b2-a073-a37ba30d0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:46:58.995557 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:58.995536 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d598a-2562-48b2-a073-a37ba30d0d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh\" (UID: \"ad5d598a-2562-48b2-a073-a37ba30d0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:46:59.071463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:59.071381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:46:59.346527 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:46:59.346494 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh"] Sep 10 13:46:59.348042 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:46:59.348013 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5d598a_2562_48b2_a073_a37ba30d0d6b.slice/crio-b7dc494ecb1db7f41d2467eea18c1508c8097f166b15f12327095d90d289f9ed WatchSource:0}: Error finding container b7dc494ecb1db7f41d2467eea18c1508c8097f166b15f12327095d90d289f9ed: Status 404 returned error can't find the container with id b7dc494ecb1db7f41d2467eea18c1508c8097f166b15f12327095d90d289f9ed Sep 10 13:47:00.283405 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:00.283311 2568 generic.go:356] "Generic (PLEG): container finished" podID="ad5d598a-2562-48b2-a073-a37ba30d0d6b" containerID="f833633ef3e384ee5b58efa3fa94bab6aed7e8e251284e97d4e351930300777e" exitCode=0 Sep 10 13:47:00.283405 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:00.283355 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" event={"ID":"ad5d598a-2562-48b2-a073-a37ba30d0d6b","Type":"ContainerDied","Data":"f833633ef3e384ee5b58efa3fa94bab6aed7e8e251284e97d4e351930300777e"} Sep 10 13:47:00.283830 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:00.283423 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" event={"ID":"ad5d598a-2562-48b2-a073-a37ba30d0d6b","Type":"ContainerStarted","Data":"b7dc494ecb1db7f41d2467eea18c1508c8097f166b15f12327095d90d289f9ed"} Sep 10 13:47:01.430035 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:01.430003 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:47:01.515590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:01.515554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d598a-2562-48b2-a073-a37ba30d0d6b-maas-api-service-ca\") pod \"ad5d598a-2562-48b2-a073-a37ba30d0d6b\" (UID: \"ad5d598a-2562-48b2-a073-a37ba30d0d6b\") " Sep 10 13:47:01.515930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:01.515905 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5d598a-2562-48b2-a073-a37ba30d0d6b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad5d598a-2562-48b2-a073-a37ba30d0d6b" (UID: "ad5d598a-2562-48b2-a073-a37ba30d0d6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:01.616546 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:01.616456 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d598a-2562-48b2-a073-a37ba30d0d6b-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:02.296152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:02.296123 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" Sep 10 13:47:02.296333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:02.296157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh" event={"ID":"ad5d598a-2562-48b2-a073-a37ba30d0d6b","Type":"ContainerDied","Data":"b7dc494ecb1db7f41d2467eea18c1508c8097f166b15f12327095d90d289f9ed"} Sep 10 13:47:02.296333 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:02.296202 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7dc494ecb1db7f41d2467eea18c1508c8097f166b15f12327095d90d289f9ed" Sep 10 13:47:05.437683 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.437645 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb"] Sep 10 13:47:05.438498 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.438459 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" podUID="6fe9dd9b-213b-4f49-89ee-3fe805a3d245" containerName="payload-pre-processing" containerID="cri-o://b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff" gracePeriod=30 Sep 10 13:47:05.655797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.655728 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh"] Sep 10 13:47:05.659875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.659838 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e325272-6gvrh"] Sep 10 13:47:05.712328 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.712300 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:47:05.756133 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.755835 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bw72\" (UniqueName: \"kubernetes.io/projected/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-kube-api-access-8bw72\") pod \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " Sep 10 13:47:05.756133 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.756025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-plugins-config-volume\") pod \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\" (UID: \"6fe9dd9b-213b-4f49-89ee-3fe805a3d245\") " Sep 10 13:47:05.756932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.756900 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fe9dd9b-213b-4f49-89ee-3fe805a3d245" (UID: "6fe9dd9b-213b-4f49-89ee-3fe805a3d245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:05.760443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.760413 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-kube-api-access-8bw72" (OuterVolumeSpecName: "kube-api-access-8bw72") pod "6fe9dd9b-213b-4f49-89ee-3fe805a3d245" (UID: "6fe9dd9b-213b-4f49-89ee-3fe805a3d245"). InnerVolumeSpecName "kube-api-access-8bw72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:05.789399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.789357 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5"] Sep 10 13:47:05.789688 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.789657 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" containerID="cri-o://5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b" gracePeriod=30 Sep 10 13:47:05.863517 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.857587 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bw72\" (UniqueName: \"kubernetes.io/projected/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-kube-api-access-8bw72\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:05.863517 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:05.857670 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe9dd9b-213b-4f49-89ee-3fe805a3d245-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:06.322182 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.322143 2568 generic.go:356] "Generic (PLEG): container finished" podID="6fe9dd9b-213b-4f49-89ee-3fe805a3d245" containerID="b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff" exitCode=0 Sep 10 13:47:06.322371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.322189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" event={"ID":"6fe9dd9b-213b-4f49-89ee-3fe805a3d245","Type":"ContainerDied","Data":"b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff"} Sep 10 13:47:06.322371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.322201 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" Sep 10 13:47:06.322371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.322270 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb" event={"ID":"6fe9dd9b-213b-4f49-89ee-3fe805a3d245","Type":"ContainerDied","Data":"6fc06189a2b158e9216524bc0b331a8e8b01bf3055e3b1aa4d0a962af79912ce"} Sep 10 13:47:06.322371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.322294 2568 scope.go:117] "RemoveContainer" containerID="b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff" Sep 10 13:47:06.331861 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.331835 2568 scope.go:117] "RemoveContainer" containerID="b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff" Sep 10 13:47:06.332191 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:47:06.332170 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff\": container with ID starting with b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff not found: ID does not exist" containerID="b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff" Sep 10 13:47:06.332256 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.332199 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff"} err="failed to get container status \"b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff\": rpc error: code = NotFound desc = could not find container \"b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff\": container with ID starting with b9e691cfdbd8bd567020c8080516f3e681a2290be17cfc0038f72224fb7d05ff not found: ID does not exist" Sep 10 13:47:06.353849 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.353805 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb"] Sep 10 13:47:06.368737 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.368703 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e325272-6d7ff4675d-txbkb"] Sep 10 13:47:06.426246 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.426208 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe9dd9b-213b-4f49-89ee-3fe805a3d245" path="/var/lib/kubelet/pods/6fe9dd9b-213b-4f49-89ee-3fe805a3d245/volumes" Sep 10 13:47:06.426709 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.426687 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5d598a-2562-48b2-a073-a37ba30d0d6b" path="/var/lib/kubelet/pods/ad5d598a-2562-48b2-a073-a37ba30d0d6b/volumes" Sep 10 13:47:06.705788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.705694 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn"] Sep 10 13:47:06.706233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.706128 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5d598a-2562-48b2-a073-a37ba30d0d6b" containerName="revoke-keys" Sep 10 13:47:06.706233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.706144 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5d598a-2562-48b2-a073-a37ba30d0d6b" containerName="revoke-keys" Sep 10 13:47:06.706233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.706154 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe9dd9b-213b-4f49-89ee-3fe805a3d245" containerName="payload-pre-processing" Sep 10 13:47:06.706233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.706159 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe9dd9b-213b-4f49-89ee-3fe805a3d245" containerName="payload-pre-processing" Sep 10 13:47:06.706233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.706234 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe9dd9b-213b-4f49-89ee-3fe805a3d245" containerName="payload-pre-processing" Sep 10 13:47:06.706499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.706245 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5d598a-2562-48b2-a073-a37ba30d0d6b" containerName="revoke-keys" Sep 10 13:47:06.710790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.710765 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.716265 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.716241 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-10e9e4e8-openshift-default-dockercfg-tdc59\"" Sep 10 13:47:06.727983 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.727952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn"] Sep 10 13:47:06.766262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-credential-socket\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-envoy\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766313 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfql\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-kube-api-access-jjfql\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-socket\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766666 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766451 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-certs\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766666 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766538 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-data\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766666 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-token\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766666 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-podinfo\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.766666 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.766662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istiod-ca-cert\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.867978 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.867932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-envoy\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868193 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.867986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfql\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-kube-api-access-jjfql\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868193 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-socket\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868193 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-certs\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-data\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-token\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868293 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-podinfo\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istiod-ca-cert\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868647 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868471 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-socket\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868647 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868539 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-certs\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-data\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868660 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-credential-socket\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.868938 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.868919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-credential-socket\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.869070 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.869050 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istiod-ca-cert\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.870475 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.870454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-envoy\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.870824 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.870805 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-podinfo\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.877125 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.877092 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-token\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:06.877718 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:06.877701 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfql\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-kube-api-access-jjfql\") pod \"e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:07.024535 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:07.024490 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:07.312659 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:07.312623 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn"] Sep 10 13:47:07.313862 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:47:07.313829 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdfdfda_5d7b_4ba5_af01_1c747f18ec68.slice/crio-bb286093065c09c9a7d4db969a399bae2e859bb2e708a7590aae9e75e1b8fcee WatchSource:0}: Error finding container bb286093065c09c9a7d4db969a399bae2e859bb2e708a7590aae9e75e1b8fcee: Status 404 returned error can't find the container with id bb286093065c09c9a7d4db969a399bae2e859bb2e708a7590aae9e75e1b8fcee Sep 10 13:47:07.316417 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:07.316384 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:47:07.316481 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:07.316457 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:47:07.316524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:07.316487 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:47:07.330149 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:07.330108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" event={"ID":"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68","Type":"ContainerStarted","Data":"bb286093065c09c9a7d4db969a399bae2e859bb2e708a7590aae9e75e1b8fcee"} Sep 10 13:47:08.338448 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:08.338400 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" event={"ID":"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68","Type":"ContainerStarted","Data":"59db1882145891873d8393f1c1396a1f436fa0886b09c1cb1b6ec5d3c684a65a"} Sep 10 13:47:08.363054 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:08.362989 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podStartSLOduration=2.362968981 podStartE2EDuration="2.362968981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:08.360709099 +0000 UTC m=+1106.429709088" watchObservedRunningTime="2026-09-10 13:47:08.362968981 +0000 UTC m=+1106.431969029" Sep 10 13:47:09.025687 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:09.025641 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:09.031047 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:09.031015 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:09.343831 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:09.343739 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:09.344886 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:09.344867 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:11.042630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.042606 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:47:11.111790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111695 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7bxn\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-kube-api-access-c7bxn\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.111790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111747 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c7b888-5bb8-48be-85b5-e84922f87524-istiod-ca-cert\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.111790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111771 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-certs\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111815 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-credential-socket\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111836 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-envoy\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111862 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-socket\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.111957 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c7b888-5bb8-48be-85b5-e84922f87524-istio-podinfo\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112016 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-istio-token\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112081 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112056 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-data\") pod \"42c7b888-5bb8-48be-85b5-e84922f87524\" (UID: \"42c7b888-5bb8-48be-85b5-e84922f87524\") " Sep 10 13:47:11.112396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112133 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:11.112396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112153 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:11.112396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112254 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c7b888-5bb8-48be-85b5-e84922f87524-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:11.112396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112261 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:11.112645 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112467 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-data" (OuterVolumeSpecName: "istio-data") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:11.112645 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112473 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c7b888-5bb8-48be-85b5-e84922f87524-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.112645 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112525 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.112645 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112539 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.112645 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.112553 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.114148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.114123 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42c7b888-5bb8-48be-85b5-e84922f87524-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:47:11.114276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.114179 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:11.114276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.114194 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-istio-token" (OuterVolumeSpecName: "istio-token") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:11.114276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.114233 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-kube-api-access-c7bxn" (OuterVolumeSpecName: "kube-api-access-c7bxn") pod "42c7b888-5bb8-48be-85b5-e84922f87524" (UID: "42c7b888-5bb8-48be-85b5-e84922f87524"). InnerVolumeSpecName "kube-api-access-c7bxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:11.213796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.213755 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c7b888-5bb8-48be-85b5-e84922f87524-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.213796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.213784 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.213796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.213793 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.213796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.213802 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7bxn\" (UniqueName: \"kubernetes.io/projected/42c7b888-5bb8-48be-85b5-e84922f87524-kube-api-access-c7bxn\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.214178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.213813 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c7b888-5bb8-48be-85b5-e84922f87524-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:11.355844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.355811 2568 generic.go:356] "Generic (PLEG): container finished" podID="42c7b888-5bb8-48be-85b5-e84922f87524" containerID="5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b" exitCode=0 Sep 10 13:47:11.356020 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.355863 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" Sep 10 13:47:11.356020 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.355897 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" event={"ID":"42c7b888-5bb8-48be-85b5-e84922f87524","Type":"ContainerDied","Data":"5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b"} Sep 10 13:47:11.356020 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.355958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5" event={"ID":"42c7b888-5bb8-48be-85b5-e84922f87524","Type":"ContainerDied","Data":"ad1c776e35aad4a8de75f26fbef1bd1384b35616ebdb6a68804413b3c4f8cc8b"} Sep 10 13:47:11.356020 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.355985 2568 scope.go:117] "RemoveContainer" containerID="5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b" Sep 10 13:47:11.366506 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.366488 2568 scope.go:117] "RemoveContainer" containerID="f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61" Sep 10 13:47:11.375952 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.375924 2568 scope.go:117] "RemoveContainer" containerID="5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b" Sep 10 13:47:11.376241 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:47:11.376220 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b\": container with ID starting with 5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b not found: ID does not exist" containerID="5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b" Sep 10 13:47:11.376288 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.376253 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b"} err="failed to get container status \"5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b\": rpc error: code = NotFound desc = could not find container \"5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b\": container with ID starting with 5680d434fcc224a8745db7242f9efae95cf8e94d20153205a27d7c4a82d9008b not found: ID does not exist" Sep 10 13:47:11.376288 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.376272 2568 scope.go:117] "RemoveContainer" containerID="f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61" Sep 10 13:47:11.376506 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:47:11.376491 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61\": container with ID starting with f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61 not found: ID does not exist" containerID="f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61" Sep 10 13:47:11.376543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.376510 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61"} err="failed to get container status \"f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61\": rpc error: code = NotFound desc = could not find container \"f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61\": container with ID starting with f235c8175e7205fc7b5f30377deb0360c01486c92e356576c68c05fa993e3e61 not found: ID does not exist" Sep 10 13:47:11.389231 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.389182 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5"] Sep 10 13:47:11.394404 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.394375 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0e325272-gw-openshift-default-d7c579ccb-vs8k5"] Sep 10 13:47:11.486640 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.486587 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp"] Sep 10 13:47:11.487031 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.487017 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" Sep 10 13:47:11.487083 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.487033 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" Sep 10 13:47:11.487083 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.487052 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" Sep 10 13:47:11.487083 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.487057 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" Sep 10 13:47:11.487179 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.487125 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" Sep 10 13:47:11.487179 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.487134 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" containerName="istio-proxy" Sep 10 13:47:11.491669 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.491652 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.495912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.495887 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86fdcf\"/\"test-model-86fdcf-kserve-self-signed-certs\"" Sep 10 13:47:11.496777 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.496749 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86fdcf\"/\"openshift-service-ca.crt\"" Sep 10 13:47:11.502581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.502555 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86fdcf\"/\"kube-root-ca.crt\"" Sep 10 13:47:11.506006 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.505986 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86fdcf\"/\"default-dockercfg-fxmxr\"" Sep 10 13:47:11.506975 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.506953 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp"] Sep 10 13:47:11.518113 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.518086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-model-cache\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.518250 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.518127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tmp-dir\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.518304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.518278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-home\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.518356 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.518338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkln\" (UniqueName: \"kubernetes.io/projected/7a09dc17-9283-4204-bd3f-b8e90d65f96a-kube-api-access-wmkln\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.518462 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.518442 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tls-certs\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.518545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.518532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-dshm\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619245 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-home\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619245 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkln\" (UniqueName: \"kubernetes.io/projected/7a09dc17-9283-4204-bd3f-b8e90d65f96a-kube-api-access-wmkln\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619488 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tls-certs\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619488 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-dshm\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619634 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-model-cache\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619634 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tmp-dir\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619634 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-home\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619868 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-model-cache\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.619912 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.619882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tmp-dir\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.621673 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.621652 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-dshm\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.621989 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.621969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tls-certs\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.633736 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.633715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkln\" (UniqueName: \"kubernetes.io/projected/7a09dc17-9283-4204-bd3f-b8e90d65f96a-kube-api-access-wmkln\") pod \"test-model-86fdcf-kserve-86cb675b7f-zscqp\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:11.801327 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:11.801286 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:12.129611 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.129573 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp"] Sep 10 13:47:12.131088 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:47:12.131027 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a09dc17_9283_4204_bd3f_b8e90d65f96a.slice/crio-4035c2311fa3c25518f9f882e78c53ed5ca8eb9ff6b5d4ce4aab0b3b7c3c61da WatchSource:0}: Error finding container 4035c2311fa3c25518f9f882e78c53ed5ca8eb9ff6b5d4ce4aab0b3b7c3c61da: Status 404 returned error can't find the container with id 4035c2311fa3c25518f9f882e78c53ed5ca8eb9ff6b5d4ce4aab0b3b7c3c61da Sep 10 13:47:12.363749 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.363705 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" event={"ID":"7a09dc17-9283-4204-bd3f-b8e90d65f96a","Type":"ContainerStarted","Data":"78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1"} Sep 10 13:47:12.363969 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.363767 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" event={"ID":"7a09dc17-9283-4204-bd3f-b8e90d65f96a","Type":"ContainerStarted","Data":"4035c2311fa3c25518f9f882e78c53ed5ca8eb9ff6b5d4ce4aab0b3b7c3c61da"} Sep 10 13:47:12.388903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.388780 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" podStartSLOduration=1.38875911 podStartE2EDuration="1.38875911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:12.386470943 +0000 UTC m=+1110.455470900" watchObservedRunningTime="2026-09-10 13:47:12.38875911 +0000 UTC m=+1110.457759076" Sep 10 13:47:12.423982 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.423949 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c7b888-5bb8-48be-85b5-e84922f87524" path="/var/lib/kubelet/pods/42c7b888-5bb8-48be-85b5-e84922f87524/volumes" Sep 10 13:47:12.505870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.505830 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh"] Sep 10 13:47:12.515662 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.515633 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.519053 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.519026 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-10e9e4e8\"" Sep 10 13:47:12.519053 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.519026 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-10e9e4e8-dockercfg-bn7h2\"" Sep 10 13:47:12.525005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.524977 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh"] Sep 10 13:47:12.628983 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.628950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5064652-48c4-4198-a25d-7cf231e3df18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.629149 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.629083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2l6\" (UniqueName: \"kubernetes.io/projected/e5064652-48c4-4198-a25d-7cf231e3df18-kube-api-access-ml2l6\") pod \"payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.730609 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.730569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5064652-48c4-4198-a25d-7cf231e3df18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.730762 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.730709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2l6\" (UniqueName: \"kubernetes.io/projected/e5064652-48c4-4198-a25d-7cf231e3df18-kube-api-access-ml2l6\") pod \"payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.731294 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.731272 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5064652-48c4-4198-a25d-7cf231e3df18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.752767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.752725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2l6\" (UniqueName: \"kubernetes.io/projected/e5064652-48c4-4198-a25d-7cf231e3df18-kube-api-access-ml2l6\") pod \"payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:12.826977 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:12.826939 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:13.117651 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:13.117623 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh"] Sep 10 13:47:13.119351 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:47:13.119321 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5064652_48c4_4198_a25d_7cf231e3df18.slice/crio-5e73256a06c44a0bb7e5e0eec412f22509e9648bfe2295e337a16ab26f69031f WatchSource:0}: Error finding container 5e73256a06c44a0bb7e5e0eec412f22509e9648bfe2295e337a16ab26f69031f: Status 404 returned error can't find the container with id 5e73256a06c44a0bb7e5e0eec412f22509e9648bfe2295e337a16ab26f69031f Sep 10 13:47:13.383532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:13.383429 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" event={"ID":"e5064652-48c4-4198-a25d-7cf231e3df18","Type":"ContainerStarted","Data":"d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c"} Sep 10 13:47:13.383932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:13.383716 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:13.383932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:13.383764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" event={"ID":"e5064652-48c4-4198-a25d-7cf231e3df18","Type":"ContainerStarted","Data":"5e73256a06c44a0bb7e5e0eec412f22509e9648bfe2295e337a16ab26f69031f"} Sep 10 13:47:13.408987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:13.408937 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" podStartSLOduration=1.4089217139999999 podStartE2EDuration="1.408921714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:13.40792787 +0000 UTC m=+1111.476927836" watchObservedRunningTime="2026-09-10 13:47:13.408921714 +0000 UTC m=+1111.477921729" Sep 10 13:47:21.802366 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:21.802317 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:21.802366 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:21.802377 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:21.815736 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:21.815707 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:22.447703 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:22.447670 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:47:24.392634 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:24.392581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:33.506023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:33.505937 2568 generic.go:356] "Generic (PLEG): container finished" podID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerID="59db1882145891873d8393f1c1396a1f436fa0886b09c1cb1b6ec5d3c684a65a" exitCode=0 Sep 10 13:47:33.506421 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:33.506022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" event={"ID":"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68","Type":"ContainerDied","Data":"59db1882145891873d8393f1c1396a1f436fa0886b09c1cb1b6ec5d3c684a65a"} Sep 10 13:47:33.506526 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:33.506511 2568 scope.go:117] "RemoveContainer" containerID="59db1882145891873d8393f1c1396a1f436fa0886b09c1cb1b6ec5d3c684a65a" Sep 10 13:47:33.507761 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:33.507732 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:47:33.507830 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:33.507806 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:47:33.507867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:33.507837 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:47:34.513463 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:34.513420 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" event={"ID":"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68","Type":"ContainerStarted","Data":"aecdfc6d7dae2c80e7a19ecf4c885ca78b1ae7d6935ed85dbcfea67513d0d796"} Sep 10 13:47:35.024734 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:35.024698 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:35.025864 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:35.025835 2568 patch_prober.go:28] interesting pod/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 10 13:47:35.025958 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:35.025892 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 10 13:47:36.025000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:36.024961 2568 patch_prober.go:28] interesting pod/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 10 13:47:36.025423 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:36.025029 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 10 13:47:37.025903 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:37.025861 2568 patch_prober.go:28] interesting pod/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 10 13:47:37.026404 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:37.025929 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 10 13:47:38.025237 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:38.025199 2568 patch_prober.go:28] interesting pod/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 10 13:47:38.025474 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:38.025263 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 10 13:47:39.025695 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:39.025653 2568 patch_prober.go:28] interesting pod/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 10 13:47:39.026188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:39.025736 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 10 13:47:39.344205 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:39.344098 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:40.029239 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:40.029209 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:40.030299 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:40.030280 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:42.922188 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:42.922146 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh"] Sep 10 13:47:42.922763 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:42.922416 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" podUID="e5064652-48c4-4198-a25d-7cf231e3df18" containerName="payload-pre-processing" containerID="cri-o://d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c" gracePeriod=30 Sep 10 13:47:43.176494 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.176430 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:43.250937 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.250903 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml2l6\" (UniqueName: \"kubernetes.io/projected/e5064652-48c4-4198-a25d-7cf231e3df18-kube-api-access-ml2l6\") pod \"e5064652-48c4-4198-a25d-7cf231e3df18\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " Sep 10 13:47:43.251139 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.251072 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5064652-48c4-4198-a25d-7cf231e3df18-plugins-config-volume\") pod \"e5064652-48c4-4198-a25d-7cf231e3df18\" (UID: \"e5064652-48c4-4198-a25d-7cf231e3df18\") " Sep 10 13:47:43.251447 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.251423 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5064652-48c4-4198-a25d-7cf231e3df18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5064652-48c4-4198-a25d-7cf231e3df18" (UID: "e5064652-48c4-4198-a25d-7cf231e3df18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:43.253059 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.253032 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5064652-48c4-4198-a25d-7cf231e3df18-kube-api-access-ml2l6" (OuterVolumeSpecName: "kube-api-access-ml2l6") pod "e5064652-48c4-4198-a25d-7cf231e3df18" (UID: "e5064652-48c4-4198-a25d-7cf231e3df18"). InnerVolumeSpecName "kube-api-access-ml2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:43.351936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.351882 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2l6\" (UniqueName: \"kubernetes.io/projected/e5064652-48c4-4198-a25d-7cf231e3df18-kube-api-access-ml2l6\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:43.351936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.351920 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5064652-48c4-4198-a25d-7cf231e3df18-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:43.571623 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.571557 2568 generic.go:356] "Generic (PLEG): container finished" podID="e5064652-48c4-4198-a25d-7cf231e3df18" containerID="d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c" exitCode=0 Sep 10 13:47:43.571807 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.571629 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" Sep 10 13:47:43.571807 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.571655 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" event={"ID":"e5064652-48c4-4198-a25d-7cf231e3df18","Type":"ContainerDied","Data":"d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c"} Sep 10 13:47:43.571807 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.571711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh" event={"ID":"e5064652-48c4-4198-a25d-7cf231e3df18","Type":"ContainerDied","Data":"5e73256a06c44a0bb7e5e0eec412f22509e9648bfe2295e337a16ab26f69031f"} Sep 10 13:47:43.571807 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.571735 2568 scope.go:117] "RemoveContainer" containerID="d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c" Sep 10 13:47:43.581624 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.581579 2568 scope.go:117] "RemoveContainer" containerID="d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c" Sep 10 13:47:43.581980 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:47:43.581957 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c\": container with ID starting with d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c not found: ID does not exist" containerID="d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c" Sep 10 13:47:43.582032 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.581992 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c"} err="failed to get container status \"d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c\": rpc error: code = NotFound desc = could not find container \"d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c\": container with ID starting with d826c8020f96ec3f9aeec216fc925ff7710686c21a5b911af3111155ebcd2e0c not found: ID does not exist" Sep 10 13:47:43.599430 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.599377 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh"] Sep 10 13:47:43.606382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:43.606348 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10e9e4e8-7788984f4b-xfhxh"] Sep 10 13:47:44.377738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:44.376969 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn"] Sep 10 13:47:44.377738 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:44.377221 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" containerID="cri-o://aecdfc6d7dae2c80e7a19ecf4c885ca78b1ae7d6935ed85dbcfea67513d0d796" gracePeriod=30 Sep 10 13:47:44.425801 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:44.425767 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5064652-48c4-4198-a25d-7cf231e3df18" path="/var/lib/kubelet/pods/e5064652-48c4-4198-a25d-7cf231e3df18/volumes" Sep 10 13:47:49.611354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:49.611319 2568 generic.go:356] "Generic (PLEG): container finished" podID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerID="aecdfc6d7dae2c80e7a19ecf4c885ca78b1ae7d6935ed85dbcfea67513d0d796" exitCode=0 Sep 10 13:47:49.611811 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:49.611391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" event={"ID":"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68","Type":"ContainerDied","Data":"aecdfc6d7dae2c80e7a19ecf4c885ca78b1ae7d6935ed85dbcfea67513d0d796"} Sep 10 13:47:49.611811 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:49.611452 2568 scope.go:117] "RemoveContainer" containerID="59db1882145891873d8393f1c1396a1f436fa0886b09c1cb1b6ec5d3c684a65a" Sep 10 13:47:50.038938 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.038914 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:50.115994 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.115963 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjfql\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-kube-api-access-jjfql\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116162 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116006 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-podinfo\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116162 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116036 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-data\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116162 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116096 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-certs\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116162 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116118 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istiod-ca-cert\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116255 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-envoy\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116338 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-token\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116373 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-credential-socket\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116414 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:50.116552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116430 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:50.116552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116426 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-socket\") pod \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\" (UID: \"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68\") " Sep 10 13:47:50.116552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116500 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:50.116760 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116683 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:50.116760 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116698 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:50.116945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116925 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.117025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116950 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.117025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116966 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.117025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116979 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.117025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.116992 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.118503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.118478 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:47:50.118639 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.118514 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:50.118815 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.118791 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:47:50.118815 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.118800 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-kube-api-access-jjfql" (OuterVolumeSpecName: "kube-api-access-jjfql") pod "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" (UID: "dbdfdfda-5d7b-4ba5-af01-1c747f18ec68"). InnerVolumeSpecName "kube-api-access-jjfql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:50.218366 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.218331 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.218366 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.218364 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.218564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.218373 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfql\" (UniqueName: \"kubernetes.io/projected/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-kube-api-access-jjfql\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.218564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.218384 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:47:50.624264 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.619455 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" Sep 10 13:47:50.624264 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.619452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn" event={"ID":"dbdfdfda-5d7b-4ba5-af01-1c747f18ec68","Type":"ContainerDied","Data":"bb286093065c09c9a7d4db969a399bae2e859bb2e708a7590aae9e75e1b8fcee"} Sep 10 13:47:50.624264 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.619634 2568 scope.go:117] "RemoveContainer" containerID="aecdfc6d7dae2c80e7a19ecf4c885ca78b1ae7d6935ed85dbcfea67513d0d796" Sep 10 13:47:50.651853 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.651810 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn"] Sep 10 13:47:50.654462 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:50.654426 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-10e9e4e8-openshift-default-fc9645cb6-b48nn"] Sep 10 13:47:52.424666 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:47:52.424632 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" path="/var/lib/kubelet/pods/dbdfdfda-5d7b-4ba5-af01-1c747f18ec68/volumes" Sep 10 13:48:12.115837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.115746 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv"] Sep 10 13:48:12.116428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.116319 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" Sep 10 13:48:12.116428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.116340 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" Sep 10 13:48:12.116428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.116370 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5064652-48c4-4198-a25d-7cf231e3df18" containerName="payload-pre-processing" Sep 10 13:48:12.116428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.116382 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5064652-48c4-4198-a25d-7cf231e3df18" containerName="payload-pre-processing" Sep 10 13:48:12.116677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.116484 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5064652-48c4-4198-a25d-7cf231e3df18" containerName="payload-pre-processing" Sep 10 13:48:12.116677 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.116498 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" Sep 10 13:48:12.119304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.119282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:12.122276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.122247 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:48:12.122798 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.122776 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:48:12.162039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.162004 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv"] Sep 10 13:48:12.233499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.233456 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9be394-c7bf-40eb-ab12-fef71aa25113-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv\" (UID: \"3f9be394-c7bf-40eb-ab12-fef71aa25113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:12.334971 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.334920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9be394-c7bf-40eb-ab12-fef71aa25113-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv\" (UID: \"3f9be394-c7bf-40eb-ab12-fef71aa25113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:12.335573 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.335554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9be394-c7bf-40eb-ab12-fef71aa25113-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv\" (UID: \"3f9be394-c7bf-40eb-ab12-fef71aa25113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:12.430033 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.429952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:12.721187 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.721160 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv"] Sep 10 13:48:12.723869 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:48:12.723839 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9be394_c7bf_40eb_ab12_fef71aa25113.slice/crio-831c719823ec54189d79a44edafad47de4b37b18bfac9223ab302f766122d652 WatchSource:0}: Error finding container 831c719823ec54189d79a44edafad47de4b37b18bfac9223ab302f766122d652: Status 404 returned error can't find the container with id 831c719823ec54189d79a44edafad47de4b37b18bfac9223ab302f766122d652 Sep 10 13:48:12.742127 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:12.742087 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" event={"ID":"3f9be394-c7bf-40eb-ab12-fef71aa25113","Type":"ContainerStarted","Data":"831c719823ec54189d79a44edafad47de4b37b18bfac9223ab302f766122d652"} Sep 10 13:48:13.751080 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:13.751044 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f9be394-c7bf-40eb-ab12-fef71aa25113" containerID="432df3e8d239129aacaf21e1b94f1c0684baa061225bcdb47d56d60c0b4bb87b" exitCode=0 Sep 10 13:48:13.751461 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:13.751125 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" event={"ID":"3f9be394-c7bf-40eb-ab12-fef71aa25113","Type":"ContainerDied","Data":"432df3e8d239129aacaf21e1b94f1c0684baa061225bcdb47d56d60c0b4bb87b"} Sep 10 13:48:14.894935 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:14.894906 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:15.061998 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:15.061902 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9be394-c7bf-40eb-ab12-fef71aa25113-maas-api-service-ca\") pod \"3f9be394-c7bf-40eb-ab12-fef71aa25113\" (UID: \"3f9be394-c7bf-40eb-ab12-fef71aa25113\") " Sep 10 13:48:15.062285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:15.062261 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9be394-c7bf-40eb-ab12-fef71aa25113-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f9be394-c7bf-40eb-ab12-fef71aa25113" (UID: "3f9be394-c7bf-40eb-ab12-fef71aa25113"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:15.163499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:15.163461 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9be394-c7bf-40eb-ab12-fef71aa25113-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:15.763532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:15.763489 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" Sep 10 13:48:15.763776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:15.763496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv" event={"ID":"3f9be394-c7bf-40eb-ab12-fef71aa25113","Type":"ContainerDied","Data":"831c719823ec54189d79a44edafad47de4b37b18bfac9223ab302f766122d652"} Sep 10 13:48:15.763776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:15.763634 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831c719823ec54189d79a44edafad47de4b37b18bfac9223ab302f766122d652" Sep 10 13:48:19.628954 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:19.628911 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv"] Sep 10 13:48:19.633183 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:19.633152 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc2301a5-r76fv"] Sep 10 13:48:20.423942 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:20.423910 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9be394-c7bf-40eb-ab12-fef71aa25113" path="/var/lib/kubelet/pods/3f9be394-c7bf-40eb-ab12-fef71aa25113/volumes" Sep 10 13:48:24.473345 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:24.473308 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w"] Sep 10 13:48:24.473809 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:24.473527 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" containerID="cri-o://d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1" gracePeriod=30 Sep 10 13:48:29.738257 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.738190 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:48:29.809765 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.809732 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pmvr\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-kube-api-access-8pmvr\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.809765 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.809772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-podinfo\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.809792 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istiod-ca-cert\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.809821 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-envoy\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.809994 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-data\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810182 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810090 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-socket\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810182 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810130 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-credential-socket\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810201 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-token\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810205 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:29.810287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810232 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-certs\") pod \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\" (UID: \"e2ef1faa-d5f2-4690-9cc3-812dcbef8001\") " Sep 10 13:48:29.810287 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810261 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:29.810496 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810378 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:29.810496 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.810428 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:29.813009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.811708 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:29.813009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.811722 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.813009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.811745 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.813009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.811760 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.813009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.811773 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.813009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.812342 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-kube-api-access-8pmvr" (OuterVolumeSpecName: "kube-api-access-8pmvr") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "kube-api-access-8pmvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:29.813408 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.813331 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:29.818338 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.813784 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:48:29.818338 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.814049 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2ef1faa-d5f2-4690-9cc3-812dcbef8001" (UID: "e2ef1faa-d5f2-4690-9cc3-812dcbef8001"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:29.841906 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.841872 2568 generic.go:356] "Generic (PLEG): container finished" podID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerID="d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1" exitCode=0 Sep 10 13:48:29.842074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.841923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" event={"ID":"e2ef1faa-d5f2-4690-9cc3-812dcbef8001","Type":"ContainerDied","Data":"d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1"} Sep 10 13:48:29.842074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.841935 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" Sep 10 13:48:29.842074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.841997 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w" event={"ID":"e2ef1faa-d5f2-4690-9cc3-812dcbef8001","Type":"ContainerDied","Data":"55aaaaa8dcdde63b1c7ba7ab744a14ddea1792bbb7c121fae23c6f81f56a7e9d"} Sep 10 13:48:29.842074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.842026 2568 scope.go:117] "RemoveContainer" containerID="d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1" Sep 10 13:48:29.852363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.852343 2568 scope.go:117] "RemoveContainer" containerID="4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286" Sep 10 13:48:29.863441 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.863422 2568 scope.go:117] "RemoveContainer" containerID="d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1" Sep 10 13:48:29.863747 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:48:29.863726 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1\": container with ID starting with d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1 not found: ID does not exist" containerID="d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1" Sep 10 13:48:29.863837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.863761 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1"} err="failed to get container status \"d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1\": rpc error: code = NotFound desc = could not find container \"d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1\": container with ID starting with d5e2eea9bc9e285489f1c5ac85237c8ab5eeea70380b8dea7c088b8850c0c8a1 not found: ID does not exist" Sep 10 13:48:29.863837 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.863785 2568 scope.go:117] "RemoveContainer" containerID="4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286" Sep 10 13:48:29.864066 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:48:29.864049 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286\": container with ID starting with 4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286 not found: ID does not exist" containerID="4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286" Sep 10 13:48:29.864106 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.864073 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286"} err="failed to get container status \"4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286\": rpc error: code = NotFound desc = could not find container \"4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286\": container with ID starting with 4d45d7c50018def1d62a640d1ad5444093ca336ef47da2215ee286c85b143286 not found: ID does not exist" Sep 10 13:48:29.870985 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.870960 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w"] Sep 10 13:48:29.873428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.873402 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b1f490-openshift-default-5f7759bf98-vfs8w"] Sep 10 13:48:29.913019 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.912973 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmvr\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-kube-api-access-8pmvr\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.913019 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.913011 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.913019 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.913026 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.913274 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.913038 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:29.913274 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:29.913049 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ef1faa-d5f2-4690-9cc3-812dcbef8001-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:48:30.151142 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151056 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c"] Sep 10 13:48:30.151452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151439 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" Sep 10 13:48:30.151510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151454 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" Sep 10 13:48:30.151510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151466 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9be394-c7bf-40eb-ab12-fef71aa25113" containerName="revoke-keys" Sep 10 13:48:30.151510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151472 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9be394-c7bf-40eb-ab12-fef71aa25113" containerName="revoke-keys" Sep 10 13:48:30.151510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151498 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" Sep 10 13:48:30.151510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151504 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" Sep 10 13:48:30.151722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151566 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdfdfda-5d7b-4ba5-af01-1c747f18ec68" containerName="istio-proxy" Sep 10 13:48:30.151722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151573 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" Sep 10 13:48:30.151722 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.151583 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9be394-c7bf-40eb-ab12-fef71aa25113" containerName="revoke-keys" Sep 10 13:48:30.155894 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.155877 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.159415 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.159389 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5d0c07\"" Sep 10 13:48:30.169819 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.169790 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c"] Sep 10 13:48:30.215049 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.215015 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bcee-5af3-425b-a387-ca4c721f3479-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d0c07-5846969b97-s855c\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.215242 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.215166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmh9w\" (UniqueName: \"kubernetes.io/projected/f2d1bcee-5af3-425b-a387-ca4c721f3479-kube-api-access-mmh9w\") pod \"payload-processing-e2e-explicit-5d0c07-5846969b97-s855c\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.316720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.316677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bcee-5af3-425b-a387-ca4c721f3479-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d0c07-5846969b97-s855c\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.316941 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.316802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmh9w\" (UniqueName: \"kubernetes.io/projected/f2d1bcee-5af3-425b-a387-ca4c721f3479-kube-api-access-mmh9w\") pod \"payload-processing-e2e-explicit-5d0c07-5846969b97-s855c\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.317489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.317462 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bcee-5af3-425b-a387-ca4c721f3479-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d0c07-5846969b97-s855c\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.328243 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.328210 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmh9w\" (UniqueName: \"kubernetes.io/projected/f2d1bcee-5af3-425b-a387-ca4c721f3479-kube-api-access-mmh9w\") pod \"payload-processing-e2e-explicit-5d0c07-5846969b97-s855c\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.424688 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.424582 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" path="/var/lib/kubelet/pods/e2ef1faa-d5f2-4690-9cc3-812dcbef8001/volumes" Sep 10 13:48:30.466498 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.466456 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:30.786363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.786336 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c"] Sep 10 13:48:30.788007 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:48:30.787982 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d1bcee_5af3_425b_a387_ca4c721f3479.slice/crio-804dc807af7384e8de34414956c502f99e3f0cbed80404480f9fda991a01ba70 WatchSource:0}: Error finding container 804dc807af7384e8de34414956c502f99e3f0cbed80404480f9fda991a01ba70: Status 404 returned error can't find the container with id 804dc807af7384e8de34414956c502f99e3f0cbed80404480f9fda991a01ba70 Sep 10 13:48:30.847974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:30.847939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" event={"ID":"f2d1bcee-5af3-425b-a387-ca4c721f3479","Type":"ContainerStarted","Data":"804dc807af7384e8de34414956c502f99e3f0cbed80404480f9fda991a01ba70"} Sep 10 13:48:31.856495 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:31.856458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" event={"ID":"f2d1bcee-5af3-425b-a387-ca4c721f3479","Type":"ContainerStarted","Data":"0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d"} Sep 10 13:48:31.856936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:31.856517 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:48:31.889334 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:31.889273 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" podStartSLOduration=1.8892559370000002 podStartE2EDuration="1.889255937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:31.886019376 +0000 UTC m=+1189.955019341" watchObservedRunningTime="2026-09-10 13:48:31.889255937 +0000 UTC m=+1189.958255902" Sep 10 13:48:42.409782 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:42.409753 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:48:42.413910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:42.413886 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:48:42.417257 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:42.417235 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:48:42.422356 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:42.422328 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:48:42.864999 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:48:42.864964 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:49:41.815299 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:41.815211 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c"] Sep 10 13:49:41.815899 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:41.815431 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" podUID="f2d1bcee-5af3-425b-a387-ca4c721f3479" containerName="payload-processing" containerID="cri-o://0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d" gracePeriod=30 Sep 10 13:49:42.072283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.072221 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:49:42.107950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.107918 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bcee-5af3-425b-a387-ca4c721f3479-plugins-config-volume\") pod \"f2d1bcee-5af3-425b-a387-ca4c721f3479\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " Sep 10 13:49:42.108115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.107987 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmh9w\" (UniqueName: \"kubernetes.io/projected/f2d1bcee-5af3-425b-a387-ca4c721f3479-kube-api-access-mmh9w\") pod \"f2d1bcee-5af3-425b-a387-ca4c721f3479\" (UID: \"f2d1bcee-5af3-425b-a387-ca4c721f3479\") " Sep 10 13:49:42.108301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.108276 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d1bcee-5af3-425b-a387-ca4c721f3479-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2d1bcee-5af3-425b-a387-ca4c721f3479" (UID: "f2d1bcee-5af3-425b-a387-ca4c721f3479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:49:42.109966 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.109945 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d1bcee-5af3-425b-a387-ca4c721f3479-kube-api-access-mmh9w" (OuterVolumeSpecName: "kube-api-access-mmh9w") pod "f2d1bcee-5af3-425b-a387-ca4c721f3479" (UID: "f2d1bcee-5af3-425b-a387-ca4c721f3479"). InnerVolumeSpecName "kube-api-access-mmh9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:42.209197 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.209158 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmh9w\" (UniqueName: \"kubernetes.io/projected/f2d1bcee-5af3-425b-a387-ca4c721f3479-kube-api-access-mmh9w\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:49:42.209197 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.209193 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bcee-5af3-425b-a387-ca4c721f3479-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:49:42.236679 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.236648 2568 generic.go:356] "Generic (PLEG): container finished" podID="f2d1bcee-5af3-425b-a387-ca4c721f3479" containerID="0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d" exitCode=0 Sep 10 13:49:42.236828 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.236696 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" Sep 10 13:49:42.236828 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.236711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" event={"ID":"f2d1bcee-5af3-425b-a387-ca4c721f3479","Type":"ContainerDied","Data":"0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d"} Sep 10 13:49:42.236828 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.236753 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c" event={"ID":"f2d1bcee-5af3-425b-a387-ca4c721f3479","Type":"ContainerDied","Data":"804dc807af7384e8de34414956c502f99e3f0cbed80404480f9fda991a01ba70"} Sep 10 13:49:42.236828 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.236774 2568 scope.go:117] "RemoveContainer" containerID="0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d" Sep 10 13:49:42.246672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.246652 2568 scope.go:117] "RemoveContainer" containerID="0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d" Sep 10 13:49:42.246975 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:49:42.246944 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d\": container with ID starting with 0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d not found: ID does not exist" containerID="0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d" Sep 10 13:49:42.247038 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.246975 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d"} err="failed to get container status \"0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d\": rpc error: code = NotFound desc = could not find container \"0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d\": container with ID starting with 0a3982c70cbb3e48f754315bd4eaa2f0ec8c3641a203104f1450263e6ee7724d not found: ID does not exist" Sep 10 13:49:42.272665 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.272639 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c"] Sep 10 13:49:42.277436 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.277416 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d0c07-5846969b97-s855c"] Sep 10 13:49:42.430522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:42.430446 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d1bcee-5af3-425b-a387-ca4c721f3479" path="/var/lib/kubelet/pods/f2d1bcee-5af3-425b-a387-ca4c721f3479/volumes" Sep 10 13:49:54.056032 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.055998 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t"] Sep 10 13:49:54.056376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.056364 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d1bcee-5af3-425b-a387-ca4c721f3479" containerName="payload-processing" Sep 10 13:49:54.056376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.056376 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d1bcee-5af3-425b-a387-ca4c721f3479" containerName="payload-processing" Sep 10 13:49:54.056464 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.056399 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" Sep 10 13:49:54.056464 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.056407 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" Sep 10 13:49:54.056539 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.056476 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ef1faa-d5f2-4690-9cc3-812dcbef8001" containerName="istio-proxy" Sep 10 13:49:54.056539 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.056487 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d1bcee-5af3-425b-a387-ca4c721f3479" containerName="payload-processing" Sep 10 13:49:54.060863 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.060837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.070700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.070680 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-9aacfe-openshift-default-dockercfg-8fczf\"" Sep 10 13:49:54.082572 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.082538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t"] Sep 10 13:49:54.124524 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-socket\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124717 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-credential-socket\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124717 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ba323f-a919-43fd-9482-9fe7f698c50e-istiod-ca-cert\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124717 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124640 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-token\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124717 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-envoy\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-certs\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvlh\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-kube-api-access-slvlh\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-data\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.124945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.124875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-podinfo\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.225987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.225953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-token\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.225987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-envoy\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-certs\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvlh\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-kube-api-access-slvlh\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-data\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-podinfo\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226250 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-socket\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-credential-socket\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ba323f-a919-43fd-9482-9fe7f698c50e-istiod-ca-cert\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-certs\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226685 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-socket\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226826 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226735 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-credential-socket\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.226890 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.226866 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-data\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.227104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.227084 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ba323f-a919-43fd-9482-9fe7f698c50e-istiod-ca-cert\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.228379 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.228359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-envoy\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.228803 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.228779 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-podinfo\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.245681 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.245653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvlh\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-kube-api-access-slvlh\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.247480 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.247458 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-token\") pod \"e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.371551 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.371460 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:54.693366 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.693338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t"] Sep 10 13:49:54.695068 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:49:54.695040 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ba323f_a919_43fd_9482_9fe7f698c50e.slice/crio-100a598e293ed9d2014eb02c245b125289fcb3e97edc3415ae9f4efa4fe40283 WatchSource:0}: Error finding container 100a598e293ed9d2014eb02c245b125289fcb3e97edc3415ae9f4efa4fe40283: Status 404 returned error can't find the container with id 100a598e293ed9d2014eb02c245b125289fcb3e97edc3415ae9f4efa4fe40283 Sep 10 13:49:54.697225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.697190 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:49:54.697330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.697257 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:49:54.697330 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:54.697284 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:49:55.316341 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:55.316299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" event={"ID":"54ba323f-a919-43fd-9482-9fe7f698c50e","Type":"ContainerStarted","Data":"203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642"} Sep 10 13:49:55.316810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:55.316358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" event={"ID":"54ba323f-a919-43fd-9482-9fe7f698c50e","Type":"ContainerStarted","Data":"100a598e293ed9d2014eb02c245b125289fcb3e97edc3415ae9f4efa4fe40283"} Sep 10 13:49:55.345429 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:55.345379 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" podStartSLOduration=1.3453650449999999 podStartE2EDuration="1.345365045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:55.342237019 +0000 UTC m=+1273.411236985" watchObservedRunningTime="2026-09-10 13:49:55.345365045 +0000 UTC m=+1273.414365010" Sep 10 13:49:55.372531 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:55.372470 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:55.377706 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:55.377683 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:56.322025 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:56.321993 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:49:56.322922 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:49:56.322905 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:50:29.090472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:29.090435 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp"] Sep 10 13:50:30.589965 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:30.589926 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t"] Sep 10 13:50:30.590468 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:30.590203 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" podUID="54ba323f-a919-43fd-9482-9fe7f698c50e" containerName="istio-proxy" containerID="cri-o://203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642" gracePeriod=30 Sep 10 13:50:31.683755 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.683714 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79"] Sep 10 13:50:31.687703 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.687678 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:31.690349 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.690326 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:50:31.690486 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.690365 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:50:31.701552 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.701520 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79"] Sep 10 13:50:31.802643 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.802577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3785cc93-f0a1-4e8b-8a90-a07a988611d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79\" (UID: \"3785cc93-f0a1-4e8b-8a90-a07a988611d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:31.903854 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.903811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3785cc93-f0a1-4e8b-8a90-a07a988611d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79\" (UID: \"3785cc93-f0a1-4e8b-8a90-a07a988611d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:31.904462 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.904443 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3785cc93-f0a1-4e8b-8a90-a07a988611d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79\" (UID: \"3785cc93-f0a1-4e8b-8a90-a07a988611d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:31.998214 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:31.998172 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:32.295108 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:32.295080 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79"] Sep 10 13:50:32.296559 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:50:32.296535 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3785cc93_f0a1_4e8b_8a90_a07a988611d2.slice/crio-8a7cf38cf6f55c5172b1ad8d0f8c2e5b1c20085c426492d22ddaff0a21328e3f WatchSource:0}: Error finding container 8a7cf38cf6f55c5172b1ad8d0f8c2e5b1c20085c426492d22ddaff0a21328e3f: Status 404 returned error can't find the container with id 8a7cf38cf6f55c5172b1ad8d0f8c2e5b1c20085c426492d22ddaff0a21328e3f Sep 10 13:50:32.522349 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:32.522308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" event={"ID":"3785cc93-f0a1-4e8b-8a90-a07a988611d2","Type":"ContainerStarted","Data":"8a7cf38cf6f55c5172b1ad8d0f8c2e5b1c20085c426492d22ddaff0a21328e3f"} Sep 10 13:50:33.529246 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:33.529214 2568 generic.go:356] "Generic (PLEG): container finished" podID="3785cc93-f0a1-4e8b-8a90-a07a988611d2" containerID="1a88ce2c5c0cd0325fd1deb3d58f7490073136d62121f1d3dbe7ebd46dcd49ce" exitCode=0 Sep 10 13:50:33.529680 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:33.529298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" event={"ID":"3785cc93-f0a1-4e8b-8a90-a07a988611d2","Type":"ContainerDied","Data":"1a88ce2c5c0cd0325fd1deb3d58f7490073136d62121f1d3dbe7ebd46dcd49ce"} Sep 10 13:50:34.666218 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:34.666191 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:34.731932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:34.731895 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3785cc93-f0a1-4e8b-8a90-a07a988611d2-maas-api-service-ca\") pod \"3785cc93-f0a1-4e8b-8a90-a07a988611d2\" (UID: \"3785cc93-f0a1-4e8b-8a90-a07a988611d2\") " Sep 10 13:50:34.732248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:34.732223 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3785cc93-f0a1-4e8b-8a90-a07a988611d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3785cc93-f0a1-4e8b-8a90-a07a988611d2" (UID: "3785cc93-f0a1-4e8b-8a90-a07a988611d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:34.833422 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:34.833325 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3785cc93-f0a1-4e8b-8a90-a07a988611d2-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:35.541532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.541494 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" event={"ID":"3785cc93-f0a1-4e8b-8a90-a07a988611d2","Type":"ContainerDied","Data":"8a7cf38cf6f55c5172b1ad8d0f8c2e5b1c20085c426492d22ddaff0a21328e3f"} Sep 10 13:50:35.541532 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.541524 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79" Sep 10 13:50:35.541790 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.541547 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a7cf38cf6f55c5172b1ad8d0f8c2e5b1c20085c426492d22ddaff0a21328e3f" Sep 10 13:50:35.860538 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.860512 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:50:35.943061 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943013 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slvlh\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-kube-api-access-slvlh\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943234 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943081 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-credential-socket\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943234 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943118 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ba323f-a919-43fd-9482-9fe7f698c50e-istiod-ca-cert\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943234 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943137 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-envoy\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943368 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943348 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-certs\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943419 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943383 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:35.943419 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943391 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-podinfo\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943459 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-data\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943456 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ba323f-a919-43fd-9482-9fe7f698c50e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:35.943521 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943484 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-socket\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943711 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943642 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-data" (OuterVolumeSpecName: "istio-data") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:35.943711 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943653 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-token\") pod \"54ba323f-a919-43fd-9482-9fe7f698c50e\" (UID: \"54ba323f-a919-43fd-9482-9fe7f698c50e\") " Sep 10 13:50:35.943711 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943688 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:35.943865 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.943838 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:35.944052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.944025 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:35.944052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.944052 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:35.944246 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.944068 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:35.944246 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.944081 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:35.944246 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.944094 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ba323f-a919-43fd-9482-9fe7f698c50e-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:35.945482 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.945452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:50:35.945482 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.945457 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-kube-api-access-slvlh" (OuterVolumeSpecName: "kube-api-access-slvlh") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "kube-api-access-slvlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:35.945674 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.945536 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:35.945674 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:35.945586 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-token" (OuterVolumeSpecName: "istio-token") pod "54ba323f-a919-43fd-9482-9fe7f698c50e" (UID: "54ba323f-a919-43fd-9482-9fe7f698c50e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:36.045467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.045413 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:36.045467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.045460 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slvlh\" (UniqueName: \"kubernetes.io/projected/54ba323f-a919-43fd-9482-9fe7f698c50e-kube-api-access-slvlh\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:36.045467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.045470 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:36.045467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.045480 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ba323f-a919-43fd-9482-9fe7f698c50e-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:36.548787 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.548749 2568 generic.go:356] "Generic (PLEG): container finished" podID="54ba323f-a919-43fd-9482-9fe7f698c50e" containerID="203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642" exitCode=0 Sep 10 13:50:36.548992 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.548807 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" Sep 10 13:50:36.548992 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.548838 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" event={"ID":"54ba323f-a919-43fd-9482-9fe7f698c50e","Type":"ContainerDied","Data":"203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642"} Sep 10 13:50:36.548992 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.548907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t" event={"ID":"54ba323f-a919-43fd-9482-9fe7f698c50e","Type":"ContainerDied","Data":"100a598e293ed9d2014eb02c245b125289fcb3e97edc3415ae9f4efa4fe40283"} Sep 10 13:50:36.548992 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.548940 2568 scope.go:117] "RemoveContainer" containerID="203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642" Sep 10 13:50:36.558468 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.558445 2568 scope.go:117] "RemoveContainer" containerID="203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642" Sep 10 13:50:36.558770 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:50:36.558750 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642\": container with ID starting with 203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642 not found: ID does not exist" containerID="203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642" Sep 10 13:50:36.558854 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.558780 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642"} err="failed to get container status \"203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642\": rpc error: code = NotFound desc = could not find container \"203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642\": container with ID starting with 203c73d1a1964796e7e21aedc3843809eefe6b009767626e402a569b49939642 not found: ID does not exist" Sep 10 13:50:36.571842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.571812 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t"] Sep 10 13:50:36.577104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:36.577078 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-9aacfe-openshift-default-6876c5d58b-7m87t"] Sep 10 13:50:38.424286 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:38.424253 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ba323f-a919-43fd-9482-9fe7f698c50e" path="/var/lib/kubelet/pods/54ba323f-a919-43fd-9482-9fe7f698c50e/volumes" Sep 10 13:50:42.656490 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:42.656445 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2"] Sep 10 13:50:42.656971 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:42.656773 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" podUID="43de254e-4752-45df-81af-3336099b3d44" containerName="payload-pre-processing" containerID="cri-o://88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb" gracePeriod=30 Sep 10 13:50:42.897266 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:42.897237 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79"] Sep 10 13:50:42.900533 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:42.900505 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-86fdcf-r5f79"] Sep 10 13:50:42.912892 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:42.912866 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:50:43.007925 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.007887 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfbrh\" (UniqueName: \"kubernetes.io/projected/43de254e-4752-45df-81af-3336099b3d44-kube-api-access-pfbrh\") pod \"43de254e-4752-45df-81af-3336099b3d44\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " Sep 10 13:50:43.007925 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.007926 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de254e-4752-45df-81af-3336099b3d44-plugins-config-volume\") pod \"43de254e-4752-45df-81af-3336099b3d44\" (UID: \"43de254e-4752-45df-81af-3336099b3d44\") " Sep 10 13:50:43.008329 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.008304 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43de254e-4752-45df-81af-3336099b3d44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43de254e-4752-45df-81af-3336099b3d44" (UID: "43de254e-4752-45df-81af-3336099b3d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:43.010130 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.010100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43de254e-4752-45df-81af-3336099b3d44-kube-api-access-pfbrh" (OuterVolumeSpecName: "kube-api-access-pfbrh") pod "43de254e-4752-45df-81af-3336099b3d44" (UID: "43de254e-4752-45df-81af-3336099b3d44"). InnerVolumeSpecName "kube-api-access-pfbrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:43.109210 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.109109 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfbrh\" (UniqueName: \"kubernetes.io/projected/43de254e-4752-45df-81af-3336099b3d44-kube-api-access-pfbrh\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:43.109210 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.109151 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de254e-4752-45df-81af-3336099b3d44-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:43.590173 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.590135 2568 generic.go:356] "Generic (PLEG): container finished" podID="43de254e-4752-45df-81af-3336099b3d44" containerID="88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb" exitCode=0 Sep 10 13:50:43.590355 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.590178 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" Sep 10 13:50:43.590355 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.590221 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" event={"ID":"43de254e-4752-45df-81af-3336099b3d44","Type":"ContainerDied","Data":"88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb"} Sep 10 13:50:43.590355 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.590272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2" event={"ID":"43de254e-4752-45df-81af-3336099b3d44","Type":"ContainerDied","Data":"5a885409bb65342b47945e5c9e66887d039b51806e109fd08d85de2d2b21a3e2"} Sep 10 13:50:43.590355 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.590300 2568 scope.go:117] "RemoveContainer" containerID="88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb" Sep 10 13:50:43.600073 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.600052 2568 scope.go:117] "RemoveContainer" containerID="88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb" Sep 10 13:50:43.600369 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:50:43.600349 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb\": container with ID starting with 88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb not found: ID does not exist" containerID="88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb" Sep 10 13:50:43.600424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.600379 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb"} err="failed to get container status \"88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb\": rpc error: code = NotFound desc = could not find container \"88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb\": container with ID starting with 88e5844fbe9b4fbf01d548c119531afba0119d45265091afc0560d7ce4f07fcb not found: ID does not exist" Sep 10 13:50:43.631398 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.629828 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2"] Sep 10 13:50:43.631398 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:43.629884 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-86fdcf-7db85489db-vdjs2"] Sep 10 13:50:44.110476 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.110402 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" podUID="7a09dc17-9283-4204-bd3f-b8e90d65f96a" containerName="main" containerID="cri-o://78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1" gracePeriod=45 Sep 10 13:50:44.363407 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.363338 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:50:44.424190 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.424158 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3785cc93-f0a1-4e8b-8a90-a07a988611d2" path="/var/lib/kubelet/pods/3785cc93-f0a1-4e8b-8a90-a07a988611d2/volumes" Sep 10 13:50:44.424522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.424508 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43de254e-4752-45df-81af-3336099b3d44" path="/var/lib/kubelet/pods/43de254e-4752-45df-81af-3336099b3d44/volumes" Sep 10 13:50:44.524269 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524234 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmkln\" (UniqueName: \"kubernetes.io/projected/7a09dc17-9283-4204-bd3f-b8e90d65f96a-kube-api-access-wmkln\") pod \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " Sep 10 13:50:44.524269 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524274 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tls-certs\") pod \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " Sep 10 13:50:44.524504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524317 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-model-cache\") pod \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " Sep 10 13:50:44.524504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524372 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-home\") pod \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " Sep 10 13:50:44.524504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524399 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tmp-dir\") pod \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " Sep 10 13:50:44.524504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524421 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-dshm\") pod \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\" (UID: \"7a09dc17-9283-4204-bd3f-b8e90d65f96a\") " Sep 10 13:50:44.524784 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524703 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a09dc17-9283-4204-bd3f-b8e90d65f96a" (UID: "7a09dc17-9283-4204-bd3f-b8e90d65f96a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:44.524784 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524719 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a09dc17-9283-4204-bd3f-b8e90d65f96a" (UID: "7a09dc17-9283-4204-bd3f-b8e90d65f96a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:44.524784 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524736 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-home" (OuterVolumeSpecName: "home") pod "7a09dc17-9283-4204-bd3f-b8e90d65f96a" (UID: "7a09dc17-9283-4204-bd3f-b8e90d65f96a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:44.524968 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524875 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-home\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:44.524968 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524896 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tmp-dir\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:44.524968 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.524910 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-model-cache\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:44.526572 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.526550 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-dshm" (OuterVolumeSpecName: "dshm") pod "7a09dc17-9283-4204-bd3f-b8e90d65f96a" (UID: "7a09dc17-9283-4204-bd3f-b8e90d65f96a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:44.526703 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.526686 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a09dc17-9283-4204-bd3f-b8e90d65f96a" (UID: "7a09dc17-9283-4204-bd3f-b8e90d65f96a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:50:44.526754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.526701 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09dc17-9283-4204-bd3f-b8e90d65f96a-kube-api-access-wmkln" (OuterVolumeSpecName: "kube-api-access-wmkln") pod "7a09dc17-9283-4204-bd3f-b8e90d65f96a" (UID: "7a09dc17-9283-4204-bd3f-b8e90d65f96a"). InnerVolumeSpecName "kube-api-access-wmkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:44.597211 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.597180 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a09dc17-9283-4204-bd3f-b8e90d65f96a" containerID="78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1" exitCode=0 Sep 10 13:50:44.597388 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.597218 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" Sep 10 13:50:44.597388 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.597261 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" event={"ID":"7a09dc17-9283-4204-bd3f-b8e90d65f96a","Type":"ContainerDied","Data":"78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1"} Sep 10 13:50:44.597388 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.597299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp" event={"ID":"7a09dc17-9283-4204-bd3f-b8e90d65f96a","Type":"ContainerDied","Data":"4035c2311fa3c25518f9f882e78c53ed5ca8eb9ff6b5d4ce4aab0b3b7c3c61da"} Sep 10 13:50:44.597388 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.597324 2568 scope.go:117] "RemoveContainer" containerID="78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1" Sep 10 13:50:44.606607 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.606581 2568 scope.go:117] "RemoveContainer" containerID="78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1" Sep 10 13:50:44.606905 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:50:44.606886 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1\": container with ID starting with 78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1 not found: ID does not exist" containerID="78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1" Sep 10 13:50:44.606976 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.606916 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1"} err="failed to get container status \"78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1\": rpc error: code = NotFound desc = could not find container \"78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1\": container with ID starting with 78b78268816e92f5d94d9daf81536ce818388189527e5b5ecdea94ac4a30eeb1 not found: ID does not exist" Sep 10 13:50:44.620259 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.620169 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp"] Sep 10 13:50:44.624871 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.624844 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-86fdcf/test-model-86fdcf-kserve-86cb675b7f-zscqp"] Sep 10 13:50:44.626098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.626078 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmkln\" (UniqueName: \"kubernetes.io/projected/7a09dc17-9283-4204-bd3f-b8e90d65f96a-kube-api-access-wmkln\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:44.626149 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.626101 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a09dc17-9283-4204-bd3f-b8e90d65f96a-tls-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:44.626149 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:44.626112 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a09dc17-9283-4204-bd3f-b8e90d65f96a-dshm\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:46.423884 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:46.423845 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a09dc17-9283-4204-bd3f-b8e90d65f96a" path="/var/lib/kubelet/pods/7a09dc17-9283-4204-bd3f-b8e90d65f96a/volumes" Sep 10 13:50:47.366967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.366933 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj"] Sep 10 13:50:47.367347 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367333 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43de254e-4752-45df-81af-3336099b3d44" containerName="payload-pre-processing" Sep 10 13:50:47.367396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367349 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43de254e-4752-45df-81af-3336099b3d44" containerName="payload-pre-processing" Sep 10 13:50:47.367396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367358 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ba323f-a919-43fd-9482-9fe7f698c50e" containerName="istio-proxy" Sep 10 13:50:47.367396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367368 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ba323f-a919-43fd-9482-9fe7f698c50e" containerName="istio-proxy" Sep 10 13:50:47.367396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367379 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3785cc93-f0a1-4e8b-8a90-a07a988611d2" containerName="revoke-keys" Sep 10 13:50:47.367396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367385 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3785cc93-f0a1-4e8b-8a90-a07a988611d2" containerName="revoke-keys" Sep 10 13:50:47.367396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367395 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09dc17-9283-4204-bd3f-b8e90d65f96a" containerName="main" Sep 10 13:50:47.367577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367401 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09dc17-9283-4204-bd3f-b8e90d65f96a" containerName="main" Sep 10 13:50:47.367577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367466 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="43de254e-4752-45df-81af-3336099b3d44" containerName="payload-pre-processing" Sep 10 13:50:47.367577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367474 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09dc17-9283-4204-bd3f-b8e90d65f96a" containerName="main" Sep 10 13:50:47.367577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367481 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ba323f-a919-43fd-9482-9fe7f698c50e" containerName="istio-proxy" Sep 10 13:50:47.367577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.367486 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3785cc93-f0a1-4e8b-8a90-a07a988611d2" containerName="revoke-keys" Sep 10 13:50:47.372021 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.372003 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:47.375250 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.375227 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:50:47.375454 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.375439 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:50:47.387066 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.387027 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj"] Sep 10 13:50:47.552350 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.552314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7788be48-20f8-43c8-9e88-b283e8c9791d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj\" (UID: \"7788be48-20f8-43c8-9e88-b283e8c9791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:47.653587 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.653503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7788be48-20f8-43c8-9e88-b283e8c9791d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj\" (UID: \"7788be48-20f8-43c8-9e88-b283e8c9791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:47.654248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.654227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7788be48-20f8-43c8-9e88-b283e8c9791d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj\" (UID: \"7788be48-20f8-43c8-9e88-b283e8c9791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:47.682651 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.682621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:47.966936 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:47.966905 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj"] Sep 10 13:50:47.969757 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:50:47.969722 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7788be48_20f8_43c8_9e88_b283e8c9791d.slice/crio-415ad57b7a360753cc27cd3c34b99be0ce927fc601bb00379342c5d584b38810 WatchSource:0}: Error finding container 415ad57b7a360753cc27cd3c34b99be0ce927fc601bb00379342c5d584b38810: Status 404 returned error can't find the container with id 415ad57b7a360753cc27cd3c34b99be0ce927fc601bb00379342c5d584b38810 Sep 10 13:50:48.624950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:48.624856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" event={"ID":"7788be48-20f8-43c8-9e88-b283e8c9791d","Type":"ContainerStarted","Data":"c1c7d72ad03bb8b972ee13651739e1edc9d5bfb805ee5ff631d964beec1fb3a7"} Sep 10 13:50:48.624950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:48.624914 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" event={"ID":"7788be48-20f8-43c8-9e88-b283e8c9791d","Type":"ContainerStarted","Data":"415ad57b7a360753cc27cd3c34b99be0ce927fc601bb00379342c5d584b38810"} Sep 10 13:50:48.642295 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:48.642241 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" podStartSLOduration=1.642222008 podStartE2EDuration="1.642222008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:48.641543203 +0000 UTC m=+1326.710543171" watchObservedRunningTime="2026-09-10 13:50:48.642222008 +0000 UTC m=+1326.711221973" Sep 10 13:50:49.631115 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:49.631077 2568 generic.go:356] "Generic (PLEG): container finished" podID="7788be48-20f8-43c8-9e88-b283e8c9791d" containerID="c1c7d72ad03bb8b972ee13651739e1edc9d5bfb805ee5ff631d964beec1fb3a7" exitCode=0 Sep 10 13:50:49.631479 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:49.631146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" event={"ID":"7788be48-20f8-43c8-9e88-b283e8c9791d","Type":"ContainerDied","Data":"c1c7d72ad03bb8b972ee13651739e1edc9d5bfb805ee5ff631d964beec1fb3a7"} Sep 10 13:50:50.768607 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:50.768568 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:50.882479 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:50.882442 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7788be48-20f8-43c8-9e88-b283e8c9791d-maas-api-service-ca\") pod \"7788be48-20f8-43c8-9e88-b283e8c9791d\" (UID: \"7788be48-20f8-43c8-9e88-b283e8c9791d\") " Sep 10 13:50:50.882862 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:50.882840 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7788be48-20f8-43c8-9e88-b283e8c9791d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7788be48-20f8-43c8-9e88-b283e8c9791d" (UID: "7788be48-20f8-43c8-9e88-b283e8c9791d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:50.983727 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:50.983693 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7788be48-20f8-43c8-9e88-b283e8c9791d-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:51.163644 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.163566 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl"] Sep 10 13:50:51.163825 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.163797 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" containerID="cri-o://e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d" gracePeriod=30 Sep 10 13:50:51.591684 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.591652 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr"] Sep 10 13:50:51.592031 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.592020 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7788be48-20f8-43c8-9e88-b283e8c9791d" containerName="revoke-keys" Sep 10 13:50:51.592088 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.592033 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7788be48-20f8-43c8-9e88-b283e8c9791d" containerName="revoke-keys" Sep 10 13:50:51.592129 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.592094 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7788be48-20f8-43c8-9e88-b283e8c9791d" containerName="revoke-keys" Sep 10 13:50:51.595311 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.595285 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:51.623100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.623068 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr"] Sep 10 13:50:51.645765 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.645735 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" Sep 10 13:50:51.645765 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.645752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj" event={"ID":"7788be48-20f8-43c8-9e88-b283e8c9791d","Type":"ContainerDied","Data":"415ad57b7a360753cc27cd3c34b99be0ce927fc601bb00379342c5d584b38810"} Sep 10 13:50:51.646010 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.645799 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415ad57b7a360753cc27cd3c34b99be0ce927fc601bb00379342c5d584b38810" Sep 10 13:50:51.690359 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.690322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7064693-69cc-488b-af5c-fc3862a791d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr\" (UID: \"c7064693-69cc-488b-af5c-fc3862a791d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:51.791544 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.791501 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7064693-69cc-488b-af5c-fc3862a791d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr\" (UID: \"c7064693-69cc-488b-af5c-fc3862a791d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:51.792152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.792130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7064693-69cc-488b-af5c-fc3862a791d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr\" (UID: \"c7064693-69cc-488b-af5c-fc3862a791d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:51.906786 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:51.906700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:52.196685 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:52.196661 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr"] Sep 10 13:50:52.198779 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:50:52.198749 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7064693_69cc_488b_af5c_fc3862a791d4.slice/crio-d3a0f0348b488e6e7265bb6218974cfaf3062df2b27a510e55e6731bd0a666db WatchSource:0}: Error finding container d3a0f0348b488e6e7265bb6218974cfaf3062df2b27a510e55e6731bd0a666db: Status 404 returned error can't find the container with id d3a0f0348b488e6e7265bb6218974cfaf3062df2b27a510e55e6731bd0a666db Sep 10 13:50:52.652928 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:52.652895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" event={"ID":"c7064693-69cc-488b-af5c-fc3862a791d4","Type":"ContainerStarted","Data":"ff67b7227febca6e144d7bcd96ed93f923af78b5fb977511c3674f5696bf950b"} Sep 10 13:50:52.653117 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:52.652947 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" event={"ID":"c7064693-69cc-488b-af5c-fc3862a791d4","Type":"ContainerStarted","Data":"d3a0f0348b488e6e7265bb6218974cfaf3062df2b27a510e55e6731bd0a666db"} Sep 10 13:50:53.659285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:53.659244 2568 generic.go:356] "Generic (PLEG): container finished" podID="c7064693-69cc-488b-af5c-fc3862a791d4" containerID="ff67b7227febca6e144d7bcd96ed93f923af78b5fb977511c3674f5696bf950b" exitCode=0 Sep 10 13:50:53.659696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:53.659294 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" event={"ID":"c7064693-69cc-488b-af5c-fc3862a791d4","Type":"ContainerDied","Data":"ff67b7227febca6e144d7bcd96ed93f923af78b5fb977511c3674f5696bf950b"} Sep 10 13:50:54.800382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:54.800354 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:54.820487 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:54.820454 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7064693-69cc-488b-af5c-fc3862a791d4-maas-api-service-ca\") pod \"c7064693-69cc-488b-af5c-fc3862a791d4\" (UID: \"c7064693-69cc-488b-af5c-fc3862a791d4\") " Sep 10 13:50:54.820810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:54.820791 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7064693-69cc-488b-af5c-fc3862a791d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7064693-69cc-488b-af5c-fc3862a791d4" (UID: "c7064693-69cc-488b-af5c-fc3862a791d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:54.921227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:54.921147 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7064693-69cc-488b-af5c-fc3862a791d4-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:55.670754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:55.670720 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" Sep 10 13:50:55.670754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:55.670737 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr" event={"ID":"c7064693-69cc-488b-af5c-fc3862a791d4","Type":"ContainerDied","Data":"d3a0f0348b488e6e7265bb6218974cfaf3062df2b27a510e55e6731bd0a666db"} Sep 10 13:50:55.670972 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:55.670781 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3a0f0348b488e6e7265bb6218974cfaf3062df2b27a510e55e6731bd0a666db" Sep 10 13:50:56.417097 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.417072 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:50:56.430737 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430650 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-certs\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.430737 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430687 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-credential-socket\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.430737 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430704 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-socket\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430757 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-token\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430782 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r4h7\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-kube-api-access-2r4h7\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430813 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-envoy\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430869 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-podinfo\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430890 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-data\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430933 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istiod-ca-cert\") pod \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\" (UID: \"56f0ae13-c5d1-47bd-b492-821ad53e08e4\") " Sep 10 13:50:56.431023 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.430992 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431157 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431175 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431324 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431345 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431366 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431382 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.431504 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.431375 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:56.433202 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.433172 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-kube-api-access-2r4h7" (OuterVolumeSpecName: "kube-api-access-2r4h7") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "kube-api-access-2r4h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:56.433202 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.433185 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:50:56.433361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.433252 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:50:56.433361 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.433314 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "56f0ae13-c5d1-47bd-b492-821ad53e08e4" (UID: "56f0ae13-c5d1-47bd-b492-821ad53e08e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:56.532767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.532729 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.532767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.532757 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.532767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.532767 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.532996 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.532778 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.532996 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.532790 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4h7\" (UniqueName: \"kubernetes.io/projected/56f0ae13-c5d1-47bd-b492-821ad53e08e4-kube-api-access-2r4h7\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.532996 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.532800 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f0ae13-c5d1-47bd-b492-821ad53e08e4-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:50:56.678875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.678839 2568 generic.go:356] "Generic (PLEG): container finished" podID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerID="e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d" exitCode=0 Sep 10 13:50:56.679104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.678874 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" event={"ID":"56f0ae13-c5d1-47bd-b492-821ad53e08e4","Type":"ContainerDied","Data":"e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d"} Sep 10 13:50:56.679104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.678908 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" Sep 10 13:50:56.679104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.678938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl" event={"ID":"56f0ae13-c5d1-47bd-b492-821ad53e08e4","Type":"ContainerDied","Data":"d25897f6b436c3af1a559840f5cfa71cf9e7ac99c36df4b85ed6513e138574ca"} Sep 10 13:50:56.679104 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.678967 2568 scope.go:117] "RemoveContainer" containerID="e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d" Sep 10 13:50:56.689029 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.689013 2568 scope.go:117] "RemoveContainer" containerID="451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752" Sep 10 13:50:56.700032 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.700003 2568 scope.go:117] "RemoveContainer" containerID="e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d" Sep 10 13:50:56.700661 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:50:56.700639 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d\": container with ID starting with e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d not found: ID does not exist" containerID="e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d" Sep 10 13:50:56.700746 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.700669 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d"} err="failed to get container status \"e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d\": rpc error: code = NotFound desc = could not find container \"e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d\": container with ID starting with e35338e872a91a1c9deb6a2bd5dbe25d1fe1e81e7b99896c11b2d71712df6d6d not found: ID does not exist" Sep 10 13:50:56.700746 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.700689 2568 scope.go:117] "RemoveContainer" containerID="451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752" Sep 10 13:50:56.701004 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:50:56.700988 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752\": container with ID starting with 451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752 not found: ID does not exist" containerID="451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752" Sep 10 13:50:56.701043 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.701008 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752"} err="failed to get container status \"451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752\": rpc error: code = NotFound desc = could not find container \"451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752\": container with ID starting with 451b70b58eda50029cc5ba3f3ddf23cd60da7f2c9eb8b9bf07194f1783266752 not found: ID does not exist" Sep 10 13:50:56.705350 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.705327 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl"] Sep 10 13:50:56.711285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:56.711264 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-86fdcf-openshift-default-d74fbbd56-j4lsl"] Sep 10 13:50:58.426205 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:58.426164 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" path="/var/lib/kubelet/pods/56f0ae13-c5d1-47bd-b492-821ad53e08e4/volumes" Sep 10 13:50:58.964165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:58.964120 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj"] Sep 10 13:50:58.966182 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:50:58.966150 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3fe902b6-pm5wj"] Sep 10 13:51:00.424805 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:00.424773 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7788be48-20f8-43c8-9e88-b283e8c9791d" path="/var/lib/kubelet/pods/7788be48-20f8-43c8-9e88-b283e8c9791d/volumes" Sep 10 13:51:03.124918 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:03.124874 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr"] Sep 10 13:51:03.127321 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:03.127297 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02fc2e-j5wxr"] Sep 10 13:51:04.423699 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:04.423664 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7064693-69cc-488b-af5c-fc3862a791d4" path="/var/lib/kubelet/pods/c7064693-69cc-488b-af5c-fc3862a791d4/volumes" Sep 10 13:51:11.155868 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:11.155785 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6"] Sep 10 13:51:11.156411 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:11.156023 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" containerID="cri-o://351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f" gracePeriod=30 Sep 10 13:51:12.062074 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062039 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w"] Sep 10 13:51:12.062432 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062419 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7064693-69cc-488b-af5c-fc3862a791d4" containerName="revoke-keys" Sep 10 13:51:12.062501 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062433 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7064693-69cc-488b-af5c-fc3862a791d4" containerName="revoke-keys" Sep 10 13:51:12.062501 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062445 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" Sep 10 13:51:12.062501 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062453 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" Sep 10 13:51:12.062501 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062468 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" Sep 10 13:51:12.062501 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062474 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" Sep 10 13:51:12.063164 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062537 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" Sep 10 13:51:12.063164 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062545 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7064693-69cc-488b-af5c-fc3862a791d4" containerName="revoke-keys" Sep 10 13:51:12.063164 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.062552 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f0ae13-c5d1-47bd-b492-821ad53e08e4" containerName="istio-proxy" Sep 10 13:51:12.067344 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.067324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.070163 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.070139 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7a11e9-openshift-default-dockercfg-hzv9q\"" Sep 10 13:51:12.080984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.080962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w"] Sep 10 13:51:12.176629 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-podinfo\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-credential-socket\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-token\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-socket\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-certs\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrltb\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-kube-api-access-vrltb\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-envoy\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.177100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.176959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-data\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278205 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-socket\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278390 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-certs\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278390 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrltb\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-kube-api-access-vrltb\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-envoy\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-data\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-podinfo\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-credential-socket\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278552 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-socket\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-token\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278940 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278702 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-certs\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.278940 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.278897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-data\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.279110 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.279087 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-credential-socket\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.279535 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.279502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.280743 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.280724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-envoy\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.280966 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.280949 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-podinfo\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.300630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.297879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-token\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.300630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.300018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrltb\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-kube-api-access-vrltb\") pod \"e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.379991 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.379904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:12.664418 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.664374 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w"] Sep 10 13:51:12.665561 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:51:12.665530 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e3f7e5_d80f_49d5_8f2a_4b7125268ea2.slice/crio-370e17b4d9211f43699e1e97c7b3fff33c816cc85b3bbe30e0a5352aa7717017 WatchSource:0}: Error finding container 370e17b4d9211f43699e1e97c7b3fff33c816cc85b3bbe30e0a5352aa7717017: Status 404 returned error can't find the container with id 370e17b4d9211f43699e1e97c7b3fff33c816cc85b3bbe30e0a5352aa7717017 Sep 10 13:51:12.668108 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.668072 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:12.668196 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.668156 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:12.668254 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.668198 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:12.769029 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.768989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" event={"ID":"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2","Type":"ContainerStarted","Data":"9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf"} Sep 10 13:51:12.769198 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.769059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" event={"ID":"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2","Type":"ContainerStarted","Data":"370e17b4d9211f43699e1e97c7b3fff33c816cc85b3bbe30e0a5352aa7717017"} Sep 10 13:51:12.790923 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:12.790864 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podStartSLOduration=0.790842136 podStartE2EDuration="790.842136ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:12.788633273 +0000 UTC m=+1350.857633237" watchObservedRunningTime="2026-09-10 13:51:12.790842136 +0000 UTC m=+1350.859842102" Sep 10 13:51:13.381032 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:13.380989 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:13.386124 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:13.386102 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:13.773967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:13.773930 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:13.774847 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:13.774828 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:16.150643 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.150588 2568 patch_prober.go:28] interesting pod/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:51:16.151061 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.150669 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:51:16.406567 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.406505 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:51:16.519391 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519358 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-certs\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519407 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-podinfo\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519574 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519521 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-data\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519684 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519588 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770d524e-88c0-4e57-95b6-a9d596155ca2-istiod-ca-cert\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519727 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519701 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-envoy\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519766 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519719 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-data" (OuterVolumeSpecName: "istio-data") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:51:16.519766 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519732 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:51:16.519878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-token\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519798 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2754\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-kube-api-access-d2754\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-credential-socket\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.519878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519867 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-socket\") pod \"770d524e-88c0-4e57-95b6-a9d596155ca2\" (UID: \"770d524e-88c0-4e57-95b6-a9d596155ca2\") " Sep 10 13:51:16.520098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.519993 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770d524e-88c0-4e57-95b6-a9d596155ca2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:51:16.520235 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.520207 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:51:16.520419 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.520399 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.520508 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.520424 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.520508 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.520437 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.520508 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.520448 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770d524e-88c0-4e57-95b6-a9d596155ca2-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.520688 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.520587 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:51:16.521993 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.521964 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-kube-api-access-d2754" (OuterVolumeSpecName: "kube-api-access-d2754") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "kube-api-access-d2754". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:51:16.522096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.522001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-token" (OuterVolumeSpecName: "istio-token") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:51:16.522193 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.522173 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:51:16.522250 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.522179 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "770d524e-88c0-4e57-95b6-a9d596155ca2" (UID: "770d524e-88c0-4e57-95b6-a9d596155ca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:51:16.620955 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.620924 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.620955 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.620949 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.620955 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.620959 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2754\" (UniqueName: \"kubernetes.io/projected/770d524e-88c0-4e57-95b6-a9d596155ca2-kube-api-access-d2754\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.621194 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.620968 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770d524e-88c0-4e57-95b6-a9d596155ca2-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.621194 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.620976 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770d524e-88c0-4e57-95b6-a9d596155ca2-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:51:16.791777 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.791743 2568 generic.go:356] "Generic (PLEG): container finished" podID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerID="351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f" exitCode=0 Sep 10 13:51:16.791945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.791792 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" Sep 10 13:51:16.791945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.791822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" event={"ID":"770d524e-88c0-4e57-95b6-a9d596155ca2","Type":"ContainerDied","Data":"351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f"} Sep 10 13:51:16.791945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.791892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6" event={"ID":"770d524e-88c0-4e57-95b6-a9d596155ca2","Type":"ContainerDied","Data":"be6bd7c120b1e2697dc26d92cd56237ba631a0921832b5abaf44ea5d9b1b115d"} Sep 10 13:51:16.791945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.791922 2568 scope.go:117] "RemoveContainer" containerID="351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f" Sep 10 13:51:16.801582 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.801565 2568 scope.go:117] "RemoveContainer" containerID="21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78" Sep 10 13:51:16.809526 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.809513 2568 scope.go:117] "RemoveContainer" containerID="351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f" Sep 10 13:51:16.809801 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:51:16.809782 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f\": container with ID starting with 351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f not found: ID does not exist" containerID="351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f" Sep 10 13:51:16.809848 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.809808 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f"} err="failed to get container status \"351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f\": rpc error: code = NotFound desc = could not find container \"351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f\": container with ID starting with 351e244f5c26f2e852455e029a1c11d892f7501df4153aa88304dc540908966f not found: ID does not exist" Sep 10 13:51:16.809848 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.809823 2568 scope.go:117] "RemoveContainer" containerID="21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78" Sep 10 13:51:16.810074 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:51:16.810056 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78\": container with ID starting with 21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78 not found: ID does not exist" containerID="21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78" Sep 10 13:51:16.810119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.810080 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78"} err="failed to get container status \"21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78\": rpc error: code = NotFound desc = could not find container \"21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78\": container with ID starting with 21191c62e035b59904bda15254d36248a5db3393e51e6ae708d8743a57cc0c78 not found: ID does not exist" Sep 10 13:51:16.839342 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.839316 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6"] Sep 10 13:51:16.857569 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:16.857536 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-02fc2e-openshift-default-66d8944f96-fdqj6"] Sep 10 13:51:17.302328 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.302292 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc"] Sep 10 13:51:17.302723 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.302693 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" Sep 10 13:51:17.302723 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.302706 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" Sep 10 13:51:17.302796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.302766 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" Sep 10 13:51:17.307256 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.307235 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.314975 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.314955 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7a11e9-dockercfg-kjc59\"" Sep 10 13:51:17.317272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.317256 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7a11e9\"" Sep 10 13:51:17.326086 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.326065 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc"] Sep 10 13:51:17.427979 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.427948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswcr\" (UniqueName: \"kubernetes.io/projected/26d9c1ca-b634-4ce3-8525-71edf23713a2-kube-api-access-bswcr\") pod \"payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.428148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.428082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d9c1ca-b634-4ce3-8525-71edf23713a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.529498 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.529460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d9c1ca-b634-4ce3-8525-71edf23713a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.529700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.529545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswcr\" (UniqueName: \"kubernetes.io/projected/26d9c1ca-b634-4ce3-8525-71edf23713a2-kube-api-access-bswcr\") pod \"payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.530233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.530207 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d9c1ca-b634-4ce3-8525-71edf23713a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.539897 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.539870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswcr\" (UniqueName: \"kubernetes.io/projected/26d9c1ca-b634-4ce3-8525-71edf23713a2-kube-api-access-bswcr\") pod \"payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.617728 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.617644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:17.899382 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:17.899351 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc"] Sep 10 13:51:17.900630 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:51:17.900593 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d9c1ca_b634_4ce3_8525_71edf23713a2.slice/crio-8f0bbd93fb869f687182eeea30b44318b516c6663e74a8bf7b8a9aacc6ba8492 WatchSource:0}: Error finding container 8f0bbd93fb869f687182eeea30b44318b516c6663e74a8bf7b8a9aacc6ba8492: Status 404 returned error can't find the container with id 8f0bbd93fb869f687182eeea30b44318b516c6663e74a8bf7b8a9aacc6ba8492 Sep 10 13:51:18.425636 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:18.425589 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" path="/var/lib/kubelet/pods/770d524e-88c0-4e57-95b6-a9d596155ca2/volumes" Sep 10 13:51:18.805119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:18.805083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" event={"ID":"26d9c1ca-b634-4ce3-8525-71edf23713a2","Type":"ContainerStarted","Data":"79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4"} Sep 10 13:51:18.805299 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:18.805143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" event={"ID":"26d9c1ca-b634-4ce3-8525-71edf23713a2","Type":"ContainerStarted","Data":"8f0bbd93fb869f687182eeea30b44318b516c6663e74a8bf7b8a9aacc6ba8492"} Sep 10 13:51:18.805299 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:18.805184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:18.829344 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:18.829293 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" podStartSLOduration=1.829276954 podStartE2EDuration="1.829276954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:18.827510615 +0000 UTC m=+1356.896510582" watchObservedRunningTime="2026-09-10 13:51:18.829276954 +0000 UTC m=+1356.898276949" Sep 10 13:51:29.813428 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:29.813397 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 13:51:29.814731 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:29.814705 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:51:29.814850 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:29.814766 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:51:30.894818 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:30.894784 2568 generic.go:356] "Generic (PLEG): container finished" podID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerID="9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf" exitCode=0 Sep 10 13:51:30.895268 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:30.894861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" event={"ID":"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2","Type":"ContainerDied","Data":"9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf"} Sep 10 13:51:30.895337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:30.895318 2568 scope.go:117] "RemoveContainer" containerID="9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf" Sep 10 13:51:30.896371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:30.896339 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:30.896440 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:30.896423 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:30.896480 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:30.896464 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:31.904047 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:31.904014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" event={"ID":"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2","Type":"ContainerStarted","Data":"bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312"} Sep 10 13:51:32.380756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:32.380720 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:32.380924 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:32.380858 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:32.380924 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:32.380896 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:33.380648 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:33.380580 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:33.381079 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:33.380674 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:34.380318 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:34.380283 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:34.380513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:34.380339 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:35.380975 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:35.380938 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:35.381371 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:35.380999 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:36.381261 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:36.381222 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:36.381672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:36.381284 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:37.380590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:37.380551 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:37.380832 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:37.380630 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:38.380529 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:38.380494 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 13:51:38.380918 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:38.380546 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 13:51:39.384046 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:39.384017 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:39.384425 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:39.384284 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:39.384932 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:39.384912 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 13:51:45.698409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.698375 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv"] Sep 10 13:51:45.698896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.698855 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" Sep 10 13:51:45.698896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.698872 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" Sep 10 13:51:45.698984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.698956 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="770d524e-88c0-4e57-95b6-a9d596155ca2" containerName="istio-proxy" Sep 10 13:51:45.703277 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.703258 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.705812 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.705791 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a00003-openshift-default-dockercfg-k4qxn\"" Sep 10 13:51:45.713944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.713926 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv"] Sep 10 13:51:45.791159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-socket\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791317 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-istio-token\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791317 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-data\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e841089e-abd4-444b-bf01-76e757472a53-istio-podinfo\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791409 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-credential-socket\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791483 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9sp\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-kube-api-access-md9sp\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791483 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791475 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-certs\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791578 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-envoy\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.791656 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.791581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e841089e-abd4-444b-bf01-76e757472a53-istiod-ca-cert\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892514 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-certs\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892733 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-envoy\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892733 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e841089e-abd4-444b-bf01-76e757472a53-istiod-ca-cert\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892733 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-socket\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-istio-token\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-data\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e841089e-abd4-444b-bf01-76e757472a53-istio-podinfo\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-credential-socket\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.892910 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9sp\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-kube-api-access-md9sp\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.893172 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.892953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-certs\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.893172 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.893115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-data\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.893282 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.893232 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-credential-socket\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.893414 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.893392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-socket\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.893697 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.893679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e841089e-abd4-444b-bf01-76e757472a53-istiod-ca-cert\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.894866 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.894845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-envoy\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.895139 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.895121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e841089e-abd4-444b-bf01-76e757472a53-istio-podinfo\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.903238 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.903191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-istio-token\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:45.903540 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:45.903513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9sp\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-kube-api-access-md9sp\") pod \"e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:46.016320 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.016294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:46.298551 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.298524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv"] Sep 10 13:51:46.300024 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:51:46.299996 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode841089e_abd4_444b_bf01_76e757472a53.slice/crio-4e8077074935314951e3a18fdac60c88c2443f49e3a09b304bd6c0c9f46d21f5 WatchSource:0}: Error finding container 4e8077074935314951e3a18fdac60c88c2443f49e3a09b304bd6c0c9f46d21f5: Status 404 returned error can't find the container with id 4e8077074935314951e3a18fdac60c88c2443f49e3a09b304bd6c0c9f46d21f5 Sep 10 13:51:46.302153 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.302119 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:46.302221 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.302184 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:46.302221 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.302216 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:46.988787 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.988747 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" event={"ID":"e841089e-abd4-444b-bf01-76e757472a53","Type":"ContainerStarted","Data":"bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0"} Sep 10 13:51:46.989172 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:46.988831 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" event={"ID":"e841089e-abd4-444b-bf01-76e757472a53","Type":"ContainerStarted","Data":"4e8077074935314951e3a18fdac60c88c2443f49e3a09b304bd6c0c9f46d21f5"} Sep 10 13:51:47.017079 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:47.017053 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:47.022700 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:47.022680 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:47.022952 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:47.022918 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podStartSLOduration=2.022906112 podStartE2EDuration="2.022906112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:47.018811309 +0000 UTC m=+1385.087811275" watchObservedRunningTime="2026-09-10 13:51:47.022906112 +0000 UTC m=+1385.091906144" Sep 10 13:51:47.995012 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:47.994980 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:47.995784 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:47.995764 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:51:51.721102 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.721062 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt"] Sep 10 13:51:51.725559 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.725536 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:51.728444 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.728422 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a00003\"" Sep 10 13:51:51.739268 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.739244 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt"] Sep 10 13:51:51.845340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.845304 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f157ad-37d8-4367-a49a-fdfa6cf5c583-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:51.845340 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.845342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8hs\" (UniqueName: \"kubernetes.io/projected/64f157ad-37d8-4367-a49a-fdfa6cf5c583-kube-api-access-lz8hs\") pod \"payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:51.946825 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.946791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f157ad-37d8-4367-a49a-fdfa6cf5c583-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:51.946825 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.946829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8hs\" (UniqueName: \"kubernetes.io/projected/64f157ad-37d8-4367-a49a-fdfa6cf5c583-kube-api-access-lz8hs\") pod \"payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:51.947512 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.947490 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f157ad-37d8-4367-a49a-fdfa6cf5c583-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:51.957346 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:51.957316 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8hs\" (UniqueName: \"kubernetes.io/projected/64f157ad-37d8-4367-a49a-fdfa6cf5c583-kube-api-access-lz8hs\") pod \"payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:52.038793 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.038721 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:52.327810 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:51:52.327780 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f157ad_37d8_4367_a49a_fdfa6cf5c583.slice/crio-4c36969fc520cecf249c9a5ee38ab92a9a1533341edfbaca397c0e9640de7da0 WatchSource:0}: Error finding container 4c36969fc520cecf249c9a5ee38ab92a9a1533341edfbaca397c0e9640de7da0: Status 404 returned error can't find the container with id 4c36969fc520cecf249c9a5ee38ab92a9a1533341edfbaca397c0e9640de7da0 Sep 10 13:51:52.328730 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.328699 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt"] Sep 10 13:51:52.763582 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.763548 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs"] Sep 10 13:51:52.767661 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.767637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.770280 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.770256 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a310dd-openshift-default-dockercfg-k7n6c\"" Sep 10 13:51:52.778945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.778926 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs"] Sep 10 13:51:52.873166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.873124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-certs\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.873442 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.873418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5346b571-363d-4a03-921b-33ee7b2f8501-istio-podinfo\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.873614 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.873583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-data\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.873780 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.873762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjr8\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-kube-api-access-lkjr8\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.873929 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.873915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-istio-token\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.874059 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.874046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5346b571-363d-4a03-921b-33ee7b2f8501-istiod-ca-cert\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.874206 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.874193 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-socket\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.874322 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.874309 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-credential-socket\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.874492 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.874479 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-envoy\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976162 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-socket\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976162 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-credential-socket\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976417 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-envoy\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976417 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-certs\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976417 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5346b571-363d-4a03-921b-33ee7b2f8501-istio-podinfo\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976417 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-data\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976417 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjr8\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-kube-api-access-lkjr8\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-istio-token\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5346b571-363d-4a03-921b-33ee7b2f8501-istiod-ca-cert\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-socket\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976594 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-credential-socket\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976920 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-certs\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.976920 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.976870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-data\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.977458 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.977124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5346b571-363d-4a03-921b-33ee7b2f8501-istiod-ca-cert\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.978911 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.978882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-envoy\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.979102 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.979081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5346b571-363d-4a03-921b-33ee7b2f8501-istio-podinfo\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.996797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.996778 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-istio-token\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:52.996915 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:52.996799 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjr8\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-kube-api-access-lkjr8\") pod \"e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:53.023297 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.023239 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" event={"ID":"64f157ad-37d8-4367-a49a-fdfa6cf5c583","Type":"ContainerStarted","Data":"652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf"} Sep 10 13:51:53.023297 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.023288 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" event={"ID":"64f157ad-37d8-4367-a49a-fdfa6cf5c583","Type":"ContainerStarted","Data":"4c36969fc520cecf249c9a5ee38ab92a9a1533341edfbaca397c0e9640de7da0"} Sep 10 13:51:53.023446 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.023340 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:51:53.062968 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.062917 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" podStartSLOduration=2.062900869 podStartE2EDuration="2.062900869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:53.058090774 +0000 UTC m=+1391.127090739" watchObservedRunningTime="2026-09-10 13:51:53.062900869 +0000 UTC m=+1391.131900839" Sep 10 13:51:53.079637 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.079610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:53.373543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.373521 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs"] Sep 10 13:51:53.375166 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:51:53.375140 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5346b571_363d_4a03_921b_33ee7b2f8501.slice/crio-be09707a3ba0ee23c256f3ce76d629f9c64fd177e37513f306928319098d760d WatchSource:0}: Error finding container be09707a3ba0ee23c256f3ce76d629f9c64fd177e37513f306928319098d760d: Status 404 returned error can't find the container with id be09707a3ba0ee23c256f3ce76d629f9c64fd177e37513f306928319098d760d Sep 10 13:51:53.377558 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.377529 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:53.377653 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.377614 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:53.377653 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:53.377645 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:51:54.031941 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:54.031906 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" event={"ID":"5346b571-363d-4a03-921b-33ee7b2f8501","Type":"ContainerStarted","Data":"2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675"} Sep 10 13:51:54.032393 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:54.031961 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" event={"ID":"5346b571-363d-4a03-921b-33ee7b2f8501","Type":"ContainerStarted","Data":"be09707a3ba0ee23c256f3ce76d629f9c64fd177e37513f306928319098d760d"} Sep 10 13:51:54.056901 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:54.056852 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podStartSLOduration=2.056837621 podStartE2EDuration="2.056837621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:54.054237301 +0000 UTC m=+1392.123237282" watchObservedRunningTime="2026-09-10 13:51:54.056837621 +0000 UTC m=+1392.125837586" Sep 10 13:51:54.080288 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:54.080259 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:54.085092 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:54.085070 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:55.037649 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:55.037622 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:51:55.038502 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:51:55.038484 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:52:04.034561 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:04.034527 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 13:52:08.112707 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:08.112671 2568 generic.go:356] "Generic (PLEG): container finished" podID="e841089e-abd4-444b-bf01-76e757472a53" containerID="bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0" exitCode=0 Sep 10 13:52:08.113128 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:08.112745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" event={"ID":"e841089e-abd4-444b-bf01-76e757472a53","Type":"ContainerDied","Data":"bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0"} Sep 10 13:52:08.113189 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:08.113174 2568 scope.go:117] "RemoveContainer" containerID="bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0" Sep 10 13:52:08.114416 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:08.114389 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:08.114471 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:08.114458 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:08.114507 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:08.114491 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:09.120065 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:09.120023 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" event={"ID":"e841089e-abd4-444b-bf01-76e757472a53","Type":"ContainerStarted","Data":"492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c"} Sep 10 13:52:10.017697 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:10.017667 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:52:10.017941 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:10.017738 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 10 13:52:10.018052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:10.018035 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 10 13:52:11.017123 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:11.017085 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 10 13:52:11.017510 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:11.017146 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 10 13:52:12.017274 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:12.017231 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 10 13:52:12.017752 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:12.017297 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 10 13:52:13.017522 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:13.017488 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 10 13:52:13.017935 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:13.017543 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 10 13:52:14.017195 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.017166 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 10 13:52:14.017323 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.017242 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 10 13:52:14.157836 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.157754 2568 generic.go:356] "Generic (PLEG): container finished" podID="5346b571-363d-4a03-921b-33ee7b2f8501" containerID="2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675" exitCode=0 Sep 10 13:52:14.158207 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.157827 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" event={"ID":"5346b571-363d-4a03-921b-33ee7b2f8501","Type":"ContainerDied","Data":"2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675"} Sep 10 13:52:14.158277 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.158262 2568 scope.go:117] "RemoveContainer" containerID="2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675" Sep 10 13:52:14.159423 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.159392 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:14.159540 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.159480 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:14.159540 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:14.159528 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:15.021216 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:15.021184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:52:15.021487 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:15.021463 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:52:15.022201 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:15.022182 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 13:52:15.166761 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:15.166728 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" event={"ID":"5346b571-363d-4a03-921b-33ee7b2f8501","Type":"ContainerStarted","Data":"74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7"} Sep 10 13:52:16.080503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:16.080465 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:52:16.080731 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:16.080707 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 13:52:16.080791 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:16.080759 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 13:52:17.080775 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:17.080738 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 13:52:17.081175 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:17.080796 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 13:52:18.080975 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:18.080938 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 13:52:18.081424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:18.080997 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 13:52:19.080649 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:19.080590 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 13:52:19.080869 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:19.080675 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 13:52:20.080564 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:20.080526 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 13:52:20.080984 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:20.080587 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 13:52:21.084567 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:21.084532 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:52:21.084979 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:21.084859 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:52:21.085511 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:21.085491 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:52:31.181565 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.181526 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7"] Sep 10 13:52:31.186661 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.186634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.189285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.189266 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-713ee90e-openshift-default-dockercfg-4xw69\"" Sep 10 13:52:31.198801 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.198774 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7"] Sep 10 13:52:31.293543 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-certs\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.293761 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2jn\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-kube-api-access-cm2jn\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.293761 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293658 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-data\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.293887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-credential-socket\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.293887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293796 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-envoy\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.293887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293855 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-socket\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.294012 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293923 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istiod-ca-cert\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.294012 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.293975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-podinfo\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.294114 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.294038 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-token\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.394870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.394826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-credential-socket\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.394870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.394872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-envoy\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.394909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-socket\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.394930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istiod-ca-cert\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.394959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-podinfo\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.394986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-token\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395204 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-certs\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2jn\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-kube-api-access-cm2jn\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-credential-socket\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-data\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395567 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395403 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-socket\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395567 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-certs\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395771 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istiod-ca-cert\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.395991 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.395972 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-data\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.397262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.397233 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-envoy\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.397562 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.397544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-podinfo\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.413159 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.413136 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-token\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.416811 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.416785 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2jn\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-kube-api-access-cm2jn\") pod \"e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.498866 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.498827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:31.799624 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.799578 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7"] Sep 10 13:52:31.801069 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:52:31.801033 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e77edd_9d50_4388_89e9_4b60a010a6d9.slice/crio-a5d89f389d27025be618c7892e8fb96590a29d7d1bd9297d395223abfb3d3e2e WatchSource:0}: Error finding container a5d89f389d27025be618c7892e8fb96590a29d7d1bd9297d395223abfb3d3e2e: Status 404 returned error can't find the container with id a5d89f389d27025be618c7892e8fb96590a29d7d1bd9297d395223abfb3d3e2e Sep 10 13:52:31.803410 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.803369 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:31.803489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.803453 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:31.803489 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:31.803485 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:32.266855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:32.266821 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" event={"ID":"f2e77edd-9d50-4388-89e9-4b60a010a6d9","Type":"ContainerStarted","Data":"0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37"} Sep 10 13:52:32.267283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:32.266884 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" event={"ID":"f2e77edd-9d50-4388-89e9-4b60a010a6d9","Type":"ContainerStarted","Data":"a5d89f389d27025be618c7892e8fb96590a29d7d1bd9297d395223abfb3d3e2e"} Sep 10 13:52:32.296257 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:32.296194 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podStartSLOduration=1.296176183 podStartE2EDuration="1.296176183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:32.293691969 +0000 UTC m=+1430.362691941" watchObservedRunningTime="2026-09-10 13:52:32.296176183 +0000 UTC m=+1430.365176147" Sep 10 13:52:32.499590 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:32.499547 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:32.506242 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:32.506213 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:33.273635 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:33.273588 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:33.274754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:33.274735 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:36.262175 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.262086 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2"] Sep 10 13:52:36.266975 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.266922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.271796 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.271769 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-713ee90e\"" Sep 10 13:52:36.272301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.272281 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-713ee90e-dockercfg-xj82n\"" Sep 10 13:52:36.283042 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.283004 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2"] Sep 10 13:52:36.446533 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.446492 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4cv\" (UniqueName: \"kubernetes.io/projected/9dc618df-8fff-4299-866f-20469a2208f7-kube-api-access-fx4cv\") pod \"payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.446719 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.446541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc618df-8fff-4299-866f-20469a2208f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.548166 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.548073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc618df-8fff-4299-866f-20469a2208f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.548349 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.548223 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4cv\" (UniqueName: \"kubernetes.io/projected/9dc618df-8fff-4299-866f-20469a2208f7-kube-api-access-fx4cv\") pod \"payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.548890 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.548869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc618df-8fff-4299-866f-20469a2208f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.558432 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.558397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4cv\" (UniqueName: \"kubernetes.io/projected/9dc618df-8fff-4299-866f-20469a2208f7-kube-api-access-fx4cv\") pod \"payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.589045 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.589003 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:36.870625 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:36.865087 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2"] Sep 10 13:52:37.302253 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:37.302214 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" event={"ID":"9dc618df-8fff-4299-866f-20469a2208f7","Type":"ContainerStarted","Data":"2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8"} Sep 10 13:52:37.302759 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:37.302269 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" event={"ID":"9dc618df-8fff-4299-866f-20469a2208f7","Type":"ContainerStarted","Data":"cddfa780e6f2c1f493149681f75ac150bb373e3e6ba15c86ba84ecb6628d9078"} Sep 10 13:52:37.302759 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:37.302354 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:37.322472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:37.322414 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" podStartSLOduration=1.322396204 podStartE2EDuration="1.322396204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:37.320965471 +0000 UTC m=+1435.389965435" watchObservedRunningTime="2026-09-10 13:52:37.322396204 +0000 UTC m=+1435.391396169" Sep 10 13:52:42.520090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:42.520055 2568 scope.go:117] "RemoveContainer" containerID="97108d68bf4dbf458e6349a0c6db318860d7e8a487e069f19db74f5ef9969df2" Sep 10 13:52:48.311167 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:48.311131 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:52:54.205125 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:52:54.205091 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e77edd_9d50_4388_89e9_4b60a010a6d9.slice/crio-0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37.scope\": RecentStats: unable to find data in memory cache]" Sep 10 13:52:54.404592 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:54.404491 2568 generic.go:356] "Generic (PLEG): container finished" podID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerID="0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37" exitCode=0 Sep 10 13:52:54.404592 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:54.404558 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" event={"ID":"f2e77edd-9d50-4388-89e9-4b60a010a6d9","Type":"ContainerDied","Data":"0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37"} Sep 10 13:52:54.405040 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:54.405025 2568 scope.go:117] "RemoveContainer" containerID="0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37" Sep 10 13:52:54.406158 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:54.406131 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:54.406215 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:54.406204 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:54.406254 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:54.406236 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:52:55.412703 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:55.412621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" event={"ID":"f2e77edd-9d50-4388-89e9-4b60a010a6d9","Type":"ContainerStarted","Data":"27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264"} Sep 10 13:52:55.499855 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:55.499820 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:52:55.500090 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:55.500059 2568 patch_prober.go:28] interesting pod/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:52:55.500252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:55.500108 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:52:56.499443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:56.499406 2568 patch_prober.go:28] interesting pod/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:52:56.499867 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:56.499475 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:52:57.499649 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:57.499593 2568 patch_prober.go:28] interesting pod/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:52:57.500042 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:57.499670 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:52:58.499714 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:58.499673 2568 patch_prober.go:28] interesting pod/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:52:58.500093 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:58.499744 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:52:59.499158 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:59.499118 2568 patch_prober.go:28] interesting pod/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:52:59.499348 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:52:59.499186 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:53:00.499256 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:00.499212 2568 patch_prober.go:28] interesting pod/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:53:00.499658 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:00.499296 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:53:01.503526 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:01.503492 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:53:01.503908 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:01.503785 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:53:01.504393 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:01.504373 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:53:42.447021 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:42.446984 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:53:42.454157 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:42.454136 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:53:42.454295 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:42.454197 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:53:42.461284 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:42.461264 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:53:42.543900 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:53:42.543875 2568 scope.go:117] "RemoveContainer" containerID="f833633ef3e384ee5b58efa3fa94bab6aed7e8e251284e97d4e351930300777e" Sep 10 13:54:04.932158 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:04.932064 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s"] Sep 10 13:54:04.935893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:04.935867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:04.938610 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:04.938583 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:54:04.938756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:04.938628 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:54:04.946746 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:04.946723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s"] Sep 10 13:54:05.044467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:05.044434 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e3912-eabb-450b-9e16-213a60307778-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s\" (UID: \"7d7e3912-eabb-450b-9e16-213a60307778\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:05.145838 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:05.145801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e3912-eabb-450b-9e16-213a60307778-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s\" (UID: \"7d7e3912-eabb-450b-9e16-213a60307778\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:05.146389 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:05.146370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e3912-eabb-450b-9e16-213a60307778-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s\" (UID: \"7d7e3912-eabb-450b-9e16-213a60307778\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:05.247154 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:05.247124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:05.535831 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:05.535749 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s"] Sep 10 13:54:05.548029 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:54:05.548001 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7e3912_eabb_450b_9e16_213a60307778.slice/crio-d9b8fc03ec52bfb4c6a446bade3d378b2a7209521f765f3e16e2c931036e26a7 WatchSource:0}: Error finding container d9b8fc03ec52bfb4c6a446bade3d378b2a7209521f765f3e16e2c931036e26a7: Status 404 returned error can't find the container with id d9b8fc03ec52bfb4c6a446bade3d378b2a7209521f765f3e16e2c931036e26a7 Sep 10 13:54:05.808743 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:05.808669 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" event={"ID":"7d7e3912-eabb-450b-9e16-213a60307778","Type":"ContainerStarted","Data":"d9b8fc03ec52bfb4c6a446bade3d378b2a7209521f765f3e16e2c931036e26a7"} Sep 10 13:54:06.815660 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:06.815622 2568 generic.go:356] "Generic (PLEG): container finished" podID="7d7e3912-eabb-450b-9e16-213a60307778" containerID="22dc109e4d696229b6784ae0afe83ee08ec6f16d100f3919b817c80a9b03dd95" exitCode=0 Sep 10 13:54:06.816052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:06.815762 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" event={"ID":"7d7e3912-eabb-450b-9e16-213a60307778","Type":"ContainerDied","Data":"22dc109e4d696229b6784ae0afe83ee08ec6f16d100f3919b817c80a9b03dd95"} Sep 10 13:54:07.956260 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:07.956237 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:08.073905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:08.073754 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e3912-eabb-450b-9e16-213a60307778-maas-api-service-ca\") pod \"7d7e3912-eabb-450b-9e16-213a60307778\" (UID: \"7d7e3912-eabb-450b-9e16-213a60307778\") " Sep 10 13:54:08.074123 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:08.074097 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7e3912-eabb-450b-9e16-213a60307778-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d7e3912-eabb-450b-9e16-213a60307778" (UID: "7d7e3912-eabb-450b-9e16-213a60307778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:54:08.074242 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:08.074229 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e3912-eabb-450b-9e16-213a60307778-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:54:08.829209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:08.829179 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" Sep 10 13:54:08.829209 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:08.829201 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s" event={"ID":"7d7e3912-eabb-450b-9e16-213a60307778","Type":"ContainerDied","Data":"d9b8fc03ec52bfb4c6a446bade3d378b2a7209521f765f3e16e2c931036e26a7"} Sep 10 13:54:08.829420 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:08.829249 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b8fc03ec52bfb4c6a446bade3d378b2a7209521f765f3e16e2c931036e26a7" Sep 10 13:54:20.560792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.560754 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2"] Sep 10 13:54:20.561343 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.560987 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" podUID="9dc618df-8fff-4299-866f-20469a2208f7" containerName="payload-processing" containerID="cri-o://2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8" gracePeriod=30 Sep 10 13:54:20.767038 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.766975 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s"] Sep 10 13:54:20.770072 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.770042 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-713ee90e-q5f5s"] Sep 10 13:54:20.826545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.826456 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:54:20.897586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.897553 2568 generic.go:356] "Generic (PLEG): container finished" podID="9dc618df-8fff-4299-866f-20469a2208f7" containerID="2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8" exitCode=0 Sep 10 13:54:20.897782 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.897630 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" Sep 10 13:54:20.897782 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.897632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" event={"ID":"9dc618df-8fff-4299-866f-20469a2208f7","Type":"ContainerDied","Data":"2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8"} Sep 10 13:54:20.897858 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.897812 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2" event={"ID":"9dc618df-8fff-4299-866f-20469a2208f7","Type":"ContainerDied","Data":"cddfa780e6f2c1f493149681f75ac150bb373e3e6ba15c86ba84ecb6628d9078"} Sep 10 13:54:20.897858 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.897841 2568 scope.go:117] "RemoveContainer" containerID="2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8" Sep 10 13:54:20.905577 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.905547 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc618df-8fff-4299-866f-20469a2208f7-plugins-config-volume\") pod \"9dc618df-8fff-4299-866f-20469a2208f7\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " Sep 10 13:54:20.905750 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.905653 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx4cv\" (UniqueName: \"kubernetes.io/projected/9dc618df-8fff-4299-866f-20469a2208f7-kube-api-access-fx4cv\") pod \"9dc618df-8fff-4299-866f-20469a2208f7\" (UID: \"9dc618df-8fff-4299-866f-20469a2208f7\") " Sep 10 13:54:20.905962 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.905939 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc618df-8fff-4299-866f-20469a2208f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9dc618df-8fff-4299-866f-20469a2208f7" (UID: "9dc618df-8fff-4299-866f-20469a2208f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:54:20.907075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.907052 2568 scope.go:117] "RemoveContainer" containerID="2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8" Sep 10 13:54:20.907426 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:54:20.907405 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8\": container with ID starting with 2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8 not found: ID does not exist" containerID="2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8" Sep 10 13:54:20.907507 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.907438 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8"} err="failed to get container status \"2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8\": rpc error: code = NotFound desc = could not find container \"2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8\": container with ID starting with 2ab2586059783d0ead487159f7d3a8dbec116b0ec83635254a6b928ac1b710b8 not found: ID does not exist" Sep 10 13:54:20.907757 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:20.907731 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc618df-8fff-4299-866f-20469a2208f7-kube-api-access-fx4cv" (OuterVolumeSpecName: "kube-api-access-fx4cv") pod "9dc618df-8fff-4299-866f-20469a2208f7" (UID: "9dc618df-8fff-4299-866f-20469a2208f7"). InnerVolumeSpecName "kube-api-access-fx4cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:54:21.006659 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:21.006590 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc618df-8fff-4299-866f-20469a2208f7-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.006659 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:21.006652 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4cv\" (UniqueName: \"kubernetes.io/projected/9dc618df-8fff-4299-866f-20469a2208f7-kube-api-access-fx4cv\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.221756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:21.221717 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2"] Sep 10 13:54:21.229000 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:21.228954 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-713ee90e-d4598db85-wrfj2"] Sep 10 13:54:22.424833 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:22.424801 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d7e3912-eabb-450b-9e16-213a60307778" path="/var/lib/kubelet/pods/7d7e3912-eabb-450b-9e16-213a60307778/volumes" Sep 10 13:54:22.425211 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:22.425145 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc618df-8fff-4299-866f-20469a2208f7" path="/var/lib/kubelet/pods/9dc618df-8fff-4299-866f-20469a2208f7/volumes" Sep 10 13:54:42.567152 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:54:42.567118 2568 scope.go:117] "RemoveContainer" containerID="432df3e8d239129aacaf21e1b94f1c0684baa061225bcdb47d56d60c0b4bb87b" Sep 10 13:55:21.278352 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:21.278301 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7"] Sep 10 13:55:21.278917 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:21.278659 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" containerID="cri-o://27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264" gracePeriod=30 Sep 10 13:55:26.533625 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.533572 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:55:26.665494 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665385 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-token\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665494 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-data\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665494 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665485 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istiod-ca-cert\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665589 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-podinfo\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665669 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm2jn\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-kube-api-access-cm2jn\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665706 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-socket\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665749 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-certs\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665789 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-envoy\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.665844 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665825 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:55:26.666147 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665879 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-credential-socket\") pod \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\" (UID: \"f2e77edd-9d50-4388-89e9-4b60a010a6d9\") " Sep 10 13:55:26.666147 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.665878 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:55:26.666147 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666104 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:55:26.666270 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666125 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:55:26.666270 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666187 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:55:26.666512 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666492 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.666563 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666519 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.666563 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666536 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.666563 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666549 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.666694 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.666565 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.667930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.667896 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:55:26.668075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.667967 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:55:26.668075 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.667983 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-kube-api-access-cm2jn" (OuterVolumeSpecName: "kube-api-access-cm2jn") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "kube-api-access-cm2jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:55:26.668276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.668260 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2e77edd-9d50-4388-89e9-4b60a010a6d9" (UID: "f2e77edd-9d50-4388-89e9-4b60a010a6d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:55:26.768144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.768104 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.768144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.768136 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2jn\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-kube-api-access-cm2jn\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.768144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.768149 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:26.768394 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:26.768160 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e77edd-9d50-4388-89e9-4b60a010a6d9-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:55:27.278363 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.278332 2568 generic.go:356] "Generic (PLEG): container finished" podID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerID="27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264" exitCode=0 Sep 10 13:55:27.278553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.278380 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" Sep 10 13:55:27.278553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.278420 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" event={"ID":"f2e77edd-9d50-4388-89e9-4b60a010a6d9","Type":"ContainerDied","Data":"27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264"} Sep 10 13:55:27.278553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.278488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7" event={"ID":"f2e77edd-9d50-4388-89e9-4b60a010a6d9","Type":"ContainerDied","Data":"a5d89f389d27025be618c7892e8fb96590a29d7d1bd9297d395223abfb3d3e2e"} Sep 10 13:55:27.278553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.278514 2568 scope.go:117] "RemoveContainer" containerID="27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264" Sep 10 13:55:27.288637 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.288618 2568 scope.go:117] "RemoveContainer" containerID="0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37" Sep 10 13:55:27.297396 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.297378 2568 scope.go:117] "RemoveContainer" containerID="27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264" Sep 10 13:55:27.297702 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:55:27.297682 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264\": container with ID starting with 27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264 not found: ID does not exist" containerID="27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264" Sep 10 13:55:27.297759 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.297713 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264"} err="failed to get container status \"27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264\": rpc error: code = NotFound desc = could not find container \"27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264\": container with ID starting with 27d5409420cc8bfbd0a7198ba201a2da830f96810fa83bef7e7a42ed67e28264 not found: ID does not exist" Sep 10 13:55:27.297759 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.297735 2568 scope.go:117] "RemoveContainer" containerID="0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37" Sep 10 13:55:27.298034 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:55:27.298012 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37\": container with ID starting with 0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37 not found: ID does not exist" containerID="0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37" Sep 10 13:55:27.298077 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.298041 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37"} err="failed to get container status \"0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37\": rpc error: code = NotFound desc = could not find container \"0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37\": container with ID starting with 0b83d65ce2d49d9c01197f2251995c8fda61459d864d8fa83e723a19fce89e37 not found: ID does not exist" Sep 10 13:55:27.304502 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.304477 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7"] Sep 10 13:55:27.309283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:27.309258 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-713ee90e-openshift-default-79cf77f7bd-nzvq7"] Sep 10 13:55:28.424756 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:28.424720 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" path="/var/lib/kubelet/pods/f2e77edd-9d50-4388-89e9-4b60a010a6d9/volumes" Sep 10 13:55:54.037513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.037471 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8"] Sep 10 13:55:54.038122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038087 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" Sep 10 13:55:54.038122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038109 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" Sep 10 13:55:54.038248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038136 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7e3912-eabb-450b-9e16-213a60307778" containerName="revoke-keys" Sep 10 13:55:54.038248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038145 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7e3912-eabb-450b-9e16-213a60307778" containerName="revoke-keys" Sep 10 13:55:54.038248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038183 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc618df-8fff-4299-866f-20469a2208f7" containerName="payload-processing" Sep 10 13:55:54.038248 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038193 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc618df-8fff-4299-866f-20469a2208f7" containerName="payload-processing" Sep 10 13:55:54.038442 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038294 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" Sep 10 13:55:54.038442 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038307 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc618df-8fff-4299-866f-20469a2208f7" containerName="payload-processing" Sep 10 13:55:54.038442 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.038325 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7e3912-eabb-450b-9e16-213a60307778" containerName="revoke-keys" Sep 10 13:55:54.041517 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.041493 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.044431 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.044405 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"openshift-service-ca.crt\"" Sep 10 13:55:54.044571 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.044407 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"kube-root-ca.crt\"" Sep 10 13:55:54.045776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.045754 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"sub-test-model-7a11e9-kserve-self-signed-certs\"" Sep 10 13:55:54.045888 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.045791 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"default-dockercfg-2jqwd\"" Sep 10 13:55:54.054933 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.054905 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8"] Sep 10 13:55:54.130551 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.130511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tls-certs\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.130755 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.130649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tmp-dir\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.130755 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.130683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpfs\" (UniqueName: \"kubernetes.io/projected/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-kube-api-access-jfpfs\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.130755 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.130726 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-model-cache\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.130871 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.130764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-home\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.130871 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.130782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-dshm\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.231914 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.231874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tmp-dir\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.231939 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpfs\" (UniqueName: \"kubernetes.io/projected/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-kube-api-access-jfpfs\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.231974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-model-cache\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.232005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-home\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232119 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.232028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-dshm\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.232177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tls-certs\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232353 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.232332 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tmp-dir\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232466 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.232383 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-model-cache\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.232466 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.232447 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-home\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.234351 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.234329 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-dshm\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.234592 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.234573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tls-certs\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.241208 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.241189 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpfs\" (UniqueName: \"kubernetes.io/projected/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-kube-api-access-jfpfs\") pod \"sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.352399 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.352306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:55:54.630297 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:54.630262 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8"] Sep 10 13:55:54.632080 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:55:54.632051 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a1b93a_76aa_4258_9aea_bcc4edd674dc.slice/crio-f85d87b5525aff5e5c84bb11cd81ff2b4917709856ced44d3b586d66a1ceaeef WatchSource:0}: Error finding container f85d87b5525aff5e5c84bb11cd81ff2b4917709856ced44d3b586d66a1ceaeef: Status 404 returned error can't find the container with id f85d87b5525aff5e5c84bb11cd81ff2b4917709856ced44d3b586d66a1ceaeef Sep 10 13:55:55.444821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:55.444729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" event={"ID":"e8a1b93a-76aa-4258-9aea-bcc4edd674dc","Type":"ContainerStarted","Data":"7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a"} Sep 10 13:55:55.444821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:55.444775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" event={"ID":"e8a1b93a-76aa-4258-9aea-bcc4edd674dc","Type":"ContainerStarted","Data":"f85d87b5525aff5e5c84bb11cd81ff2b4917709856ced44d3b586d66a1ceaeef"} Sep 10 13:55:55.469921 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:55:55.469868 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" podStartSLOduration=1.469848142 podStartE2EDuration="1.469848142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:55.46675782 +0000 UTC m=+1633.535757784" watchObservedRunningTime="2026-09-10 13:55:55.469848142 +0000 UTC m=+1633.538848104" Sep 10 13:56:04.353098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:04.353058 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:56:04.353098 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:04.353112 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:56:04.367205 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:04.367168 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:56:04.506137 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:04.506106 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:56:19.691589 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.691551 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t"] Sep 10 13:56:19.692178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.692157 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" Sep 10 13:56:19.692271 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.692181 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" Sep 10 13:56:19.692324 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.692285 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e77edd-9d50-4388-89e9-4b60a010a6d9" containerName="istio-proxy" Sep 10 13:56:19.695369 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.695348 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.698122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.698100 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ab74b3-openshift-default-dockercfg-ltbrp\"" Sep 10 13:56:19.708547 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.708517 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t"] Sep 10 13:56:19.796487 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.796442 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5629076-33c1-4042-883a-4b1e4e458fff-istio-podinfo\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.796795 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.796766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-data\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.796927 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.796811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-envoy\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.797005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.796912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5629076-33c1-4042-883a-4b1e4e458fff-istiod-ca-cert\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.797005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.796982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-socket\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.797112 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.797025 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-credential-socket\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.797167 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.797107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdms\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-kube-api-access-7qdms\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.797167 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.797145 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-istio-token\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.797272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.797175 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-certs\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898354 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-envoy\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5629076-33c1-4042-883a-4b1e4e458fff-istiod-ca-cert\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-socket\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-credential-socket\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdms\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-kube-api-access-7qdms\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898581 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898563 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-istio-token\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898897 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898591 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-certs\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.898957 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.898935 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-credential-socket\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.899085 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.899060 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-socket\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.899168 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.899143 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-certs\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.899224 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.899180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5629076-33c1-4042-883a-4b1e4e458fff-istio-podinfo\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.899272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.899226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-data\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.899376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.899353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5629076-33c1-4042-883a-4b1e4e458fff-istiod-ca-cert\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.899513 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.899496 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-data\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.900911 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.900893 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-envoy\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.901414 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.901397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5629076-33c1-4042-883a-4b1e4e458fff-istio-podinfo\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.911283 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.911255 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-istio-token\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:19.911419 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:19.911401 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdms\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-kube-api-access-7qdms\") pod \"e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:20.011250 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.011205 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:20.302430 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.302402 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t"] Sep 10 13:56:20.303395 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:56:20.303363 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5629076_33c1_4042_883a_4b1e4e458fff.slice/crio-06f64764adaafef1805a7ba083218400a8bc6170ed4e6dd481272ada57c97bf4 WatchSource:0}: Error finding container 06f64764adaafef1805a7ba083218400a8bc6170ed4e6dd481272ada57c97bf4: Status 404 returned error can't find the container with id 06f64764adaafef1805a7ba083218400a8bc6170ed4e6dd481272ada57c97bf4 Sep 10 13:56:20.305746 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.305712 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:56:20.305822 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.305783 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:56:20.305822 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.305814 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:56:20.585881 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.585786 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" event={"ID":"e5629076-33c1-4042-883a-4b1e4e458fff","Type":"ContainerStarted","Data":"de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195"} Sep 10 13:56:20.585881 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.585853 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" event={"ID":"e5629076-33c1-4042-883a-4b1e4e458fff","Type":"ContainerStarted","Data":"06f64764adaafef1805a7ba083218400a8bc6170ed4e6dd481272ada57c97bf4"} Sep 10 13:56:20.621334 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:20.621269 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podStartSLOduration=1.6212467670000001 podStartE2EDuration="1.621246767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:20.619426435 +0000 UTC m=+1658.688426400" watchObservedRunningTime="2026-09-10 13:56:20.621246767 +0000 UTC m=+1658.690246733" Sep 10 13:56:21.012301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:21.012262 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:21.018154 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:21.018121 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:21.591681 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:21.591648 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:21.592751 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:21.592730 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:24.980433 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:24.980390 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv"] Sep 10 13:56:24.984220 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:24.984197 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:24.987203 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:24.987182 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ab74b3-dockercfg-dg7kj\"" Sep 10 13:56:24.987449 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:24.987431 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ab74b3\"" Sep 10 13:56:24.998145 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:24.998119 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv"] Sep 10 13:56:25.050187 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.050151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dnj\" (UniqueName: \"kubernetes.io/projected/fe898f81-5983-4e95-8807-ae70cf775321-kube-api-access-k4dnj\") pod \"payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.050375 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.050203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe898f81-5983-4e95-8807-ae70cf775321-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.151654 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.151618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe898f81-5983-4e95-8807-ae70cf775321-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.151964 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.151752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dnj\" (UniqueName: \"kubernetes.io/projected/fe898f81-5983-4e95-8807-ae70cf775321-kube-api-access-k4dnj\") pod \"payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.152427 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.152399 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe898f81-5983-4e95-8807-ae70cf775321-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.164649 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.164623 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dnj\" (UniqueName: \"kubernetes.io/projected/fe898f81-5983-4e95-8807-ae70cf775321-kube-api-access-k4dnj\") pod \"payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.296292 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.296191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:25.575016 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.574983 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv"] Sep 10 13:56:25.577023 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:56:25.576989 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe898f81_5983_4e95_8807_ae70cf775321.slice/crio-8636c909791fb5ae392ceaa653972b8f32e411fac248f4da63d826976f55f266 WatchSource:0}: Error finding container 8636c909791fb5ae392ceaa653972b8f32e411fac248f4da63d826976f55f266: Status 404 returned error can't find the container with id 8636c909791fb5ae392ceaa653972b8f32e411fac248f4da63d826976f55f266 Sep 10 13:56:25.616808 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:25.616780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" event={"ID":"fe898f81-5983-4e95-8807-ae70cf775321","Type":"ContainerStarted","Data":"8636c909791fb5ae392ceaa653972b8f32e411fac248f4da63d826976f55f266"} Sep 10 13:56:26.624468 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:26.624430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" event={"ID":"fe898f81-5983-4e95-8807-ae70cf775321","Type":"ContainerStarted","Data":"26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812"} Sep 10 13:56:26.624948 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:26.624521 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:26.647181 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:26.647125 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" podStartSLOduration=2.647105556 podStartE2EDuration="2.647105556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:26.645724712 +0000 UTC m=+1664.714724679" watchObservedRunningTime="2026-09-10 13:56:26.647105556 +0000 UTC m=+1664.716105521" Sep 10 13:56:37.641341 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:37.633418 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:56:37.641887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:37.641436 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:56:37.641887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:37.641521 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:56:39.707091 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:39.707003 2568 generic.go:356] "Generic (PLEG): container finished" podID="e5629076-33c1-4042-883a-4b1e4e458fff" containerID="de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195" exitCode=0 Sep 10 13:56:39.707497 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:39.707083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" event={"ID":"e5629076-33c1-4042-883a-4b1e4e458fff","Type":"ContainerDied","Data":"de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195"} Sep 10 13:56:39.707551 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:39.707536 2568 scope.go:117] "RemoveContainer" containerID="de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195" Sep 10 13:56:39.708849 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:39.708817 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:56:39.708950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:39.708891 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:56:39.708950 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:39.708920 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:56:40.714958 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:40.714925 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" event={"ID":"e5629076-33c1-4042-883a-4b1e4e458fff","Type":"ContainerStarted","Data":"ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a"} Sep 10 13:56:41.011878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:41.011771 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:41.012059 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:41.012034 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 13:56:41.012112 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:41.012087 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 13:56:42.012380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:42.012342 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 13:56:42.012824 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:42.012407 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 13:56:42.604814 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:42.604777 2568 scope.go:117] "RemoveContainer" containerID="1a88ce2c5c0cd0325fd1deb3d58f7490073136d62121f1d3dbe7ebd46dcd49ce" Sep 10 13:56:43.012465 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:43.012428 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 13:56:43.012893 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:43.012487 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 13:56:44.011696 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:44.011647 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 13:56:44.011916 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:44.011723 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 13:56:45.011767 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:45.011725 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 13:56:45.012178 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:45.011787 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 13:56:46.012227 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:46.012185 2568 patch_prober.go:28] interesting pod/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 13:56:46.012672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:46.012253 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 13:56:47.015987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:47.015950 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:47.016454 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:47.016288 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:47.017122 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:47.017103 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:56:58.988804 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:58.988766 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22"] Sep 10 13:56:58.992934 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:58.992913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:58.998041 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:58.998016 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8aa282\"" Sep 10 13:56:58.998199 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:58.998139 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8aa282-dockercfg-bz746\"" Sep 10 13:56:59.007402 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.007352 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22"] Sep 10 13:56:59.111456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.111411 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b128fc9-ac3a-4426-abc4-76455e328f65-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.111672 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.111493 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjpd\" (UniqueName: \"kubernetes.io/projected/5b128fc9-ac3a-4426-abc4-76455e328f65-kube-api-access-gmjpd\") pod \"payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.212820 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.212774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b128fc9-ac3a-4426-abc4-76455e328f65-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.213018 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.212866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjpd\" (UniqueName: \"kubernetes.io/projected/5b128fc9-ac3a-4426-abc4-76455e328f65-kube-api-access-gmjpd\") pod \"payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.213671 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.213650 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b128fc9-ac3a-4426-abc4-76455e328f65-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.225170 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.225135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjpd\" (UniqueName: \"kubernetes.io/projected/5b128fc9-ac3a-4426-abc4-76455e328f65-kube-api-access-gmjpd\") pod \"payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.303670 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.303539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.585673 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.585642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22"] Sep 10 13:56:59.588535 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:56:59.588501 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b128fc9_ac3a_4426_abc4_76455e328f65.slice/crio-377f3a1d820915f77e089f70711042d721c3f66dce476503ffee74e515acd540 WatchSource:0}: Error finding container 377f3a1d820915f77e089f70711042d721c3f66dce476503ffee74e515acd540: Status 404 returned error can't find the container with id 377f3a1d820915f77e089f70711042d721c3f66dce476503ffee74e515acd540 Sep 10 13:56:59.831763 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.831725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" event={"ID":"5b128fc9-ac3a-4426-abc4-76455e328f65","Type":"ContainerStarted","Data":"66d0ed5300240496f7a430071282451b5e832a3cd9dc14a23642ac03597166eb"} Sep 10 13:56:59.831960 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.831784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" event={"ID":"5b128fc9-ac3a-4426-abc4-76455e328f65","Type":"ContainerStarted","Data":"377f3a1d820915f77e089f70711042d721c3f66dce476503ffee74e515acd540"} Sep 10 13:56:59.831960 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.831896 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:56:59.855320 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:56:59.855201 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" podStartSLOduration=1.855178318 podStartE2EDuration="1.855178318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:59.855124423 +0000 UTC m=+1697.924124389" watchObservedRunningTime="2026-09-10 13:56:59.855178318 +0000 UTC m=+1697.924178286" Sep 10 13:57:10.839233 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:10.839149 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:57:29.783834 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.783791 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg"] Sep 10 13:57:29.787679 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.787660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:29.792613 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.792573 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:57:29.792613 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.792609 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:57:29.812443 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.812409 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg"] Sep 10 13:57:29.842380 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.842335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ccd18c-453a-4223-826f-fbfffb0b3472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg\" (UID: \"08ccd18c-453a-4223-826f-fbfffb0b3472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:29.943444 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.943400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ccd18c-453a-4223-826f-fbfffb0b3472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg\" (UID: \"08ccd18c-453a-4223-826f-fbfffb0b3472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:29.944134 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:29.944115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ccd18c-453a-4223-826f-fbfffb0b3472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg\" (UID: \"08ccd18c-453a-4223-826f-fbfffb0b3472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:30.099083 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:30.098986 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:30.381052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:30.381021 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg"] Sep 10 13:57:30.383158 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:57:30.383129 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ccd18c_453a_4223_826f_fbfffb0b3472.slice/crio-f75b9d636bfc0904787e9292e1790be5a5cfc0312247e2b744d7dbee5ad13a94 WatchSource:0}: Error finding container f75b9d636bfc0904787e9292e1790be5a5cfc0312247e2b744d7dbee5ad13a94: Status 404 returned error can't find the container with id f75b9d636bfc0904787e9292e1790be5a5cfc0312247e2b744d7dbee5ad13a94 Sep 10 13:57:31.017375 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:31.017339 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" event={"ID":"08ccd18c-453a-4223-826f-fbfffb0b3472","Type":"ContainerStarted","Data":"c2d3d541c9d829068867ca578ef462cdea3718db28b086772a56dcc5f040c66f"} Sep 10 13:57:31.017896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:31.017440 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" event={"ID":"08ccd18c-453a-4223-826f-fbfffb0b3472","Type":"ContainerStarted","Data":"f75b9d636bfc0904787e9292e1790be5a5cfc0312247e2b744d7dbee5ad13a94"} Sep 10 13:57:31.057349 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:31.057300 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" podStartSLOduration=2.057283087 podStartE2EDuration="2.057283087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:57:31.053773746 +0000 UTC m=+1729.122773711" watchObservedRunningTime="2026-09-10 13:57:31.057283087 +0000 UTC m=+1729.126283051" Sep 10 13:57:32.025587 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:32.025549 2568 generic.go:356] "Generic (PLEG): container finished" podID="08ccd18c-453a-4223-826f-fbfffb0b3472" containerID="c2d3d541c9d829068867ca578ef462cdea3718db28b086772a56dcc5f040c66f" exitCode=0 Sep 10 13:57:32.026002 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:32.025593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" event={"ID":"08ccd18c-453a-4223-826f-fbfffb0b3472","Type":"ContainerDied","Data":"c2d3d541c9d829068867ca578ef462cdea3718db28b086772a56dcc5f040c66f"} Sep 10 13:57:33.174093 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:33.174066 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:33.278503 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:33.278466 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ccd18c-453a-4223-826f-fbfffb0b3472-maas-api-service-ca\") pod \"08ccd18c-453a-4223-826f-fbfffb0b3472\" (UID: \"08ccd18c-453a-4223-826f-fbfffb0b3472\") " Sep 10 13:57:33.278887 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:33.278864 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ccd18c-453a-4223-826f-fbfffb0b3472-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08ccd18c-453a-4223-826f-fbfffb0b3472" (UID: "08ccd18c-453a-4223-826f-fbfffb0b3472"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:57:33.380272 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:33.380174 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ccd18c-453a-4223-826f-fbfffb0b3472-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:57:34.038966 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:34.038937 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" Sep 10 13:57:34.039165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:34.038974 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg" event={"ID":"08ccd18c-453a-4223-826f-fbfffb0b3472","Type":"ContainerDied","Data":"f75b9d636bfc0904787e9292e1790be5a5cfc0312247e2b744d7dbee5ad13a94"} Sep 10 13:57:34.039165 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:34.039032 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f75b9d636bfc0904787e9292e1790be5a5cfc0312247e2b744d7dbee5ad13a94" Sep 10 13:57:42.633776 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:42.633743 2568 scope.go:117] "RemoveContainer" containerID="ff67b7227febca6e144d7bcd96ed93f923af78b5fb977511c3674f5696bf950b" Sep 10 13:57:42.642749 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:42.642727 2568 scope.go:117] "RemoveContainer" containerID="c1c7d72ad03bb8b972ee13651739e1edc9d5bfb805ee5ff631d964beec1fb3a7" Sep 10 13:57:45.754642 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:45.754590 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv"] Sep 10 13:57:45.755252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:45.754832 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" podUID="fe898f81-5983-4e95-8807-ae70cf775321" containerName="payload-processing" containerID="cri-o://26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812" gracePeriod=30 Sep 10 13:57:46.046875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.046847 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:57:46.054457 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.054421 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg"] Sep 10 13:57:46.062148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.062115 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab74b3-qvcvg"] Sep 10 13:57:46.117609 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.117566 2568 generic.go:356] "Generic (PLEG): container finished" podID="fe898f81-5983-4e95-8807-ae70cf775321" containerID="26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812" exitCode=0 Sep 10 13:57:46.117810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.117637 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" Sep 10 13:57:46.117810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.117669 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" event={"ID":"fe898f81-5983-4e95-8807-ae70cf775321","Type":"ContainerDied","Data":"26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812"} Sep 10 13:57:46.117810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.117731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv" event={"ID":"fe898f81-5983-4e95-8807-ae70cf775321","Type":"ContainerDied","Data":"8636c909791fb5ae392ceaa653972b8f32e411fac248f4da63d826976f55f266"} Sep 10 13:57:46.117810 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.117765 2568 scope.go:117] "RemoveContainer" containerID="26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812" Sep 10 13:57:46.128095 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.128073 2568 scope.go:117] "RemoveContainer" containerID="26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812" Sep 10 13:57:46.128453 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:57:46.128432 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812\": container with ID starting with 26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812 not found: ID does not exist" containerID="26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812" Sep 10 13:57:46.128499 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.128464 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812"} err="failed to get container status \"26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812\": rpc error: code = NotFound desc = could not find container \"26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812\": container with ID starting with 26ed11246d1a7f6451c0575965ed8ff6fd3d681d56019547a1d61d7302f04812 not found: ID does not exist" Sep 10 13:57:46.221037 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.220998 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe898f81-5983-4e95-8807-ae70cf775321-plugins-config-volume\") pod \"fe898f81-5983-4e95-8807-ae70cf775321\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " Sep 10 13:57:46.221037 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.221042 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4dnj\" (UniqueName: \"kubernetes.io/projected/fe898f81-5983-4e95-8807-ae70cf775321-kube-api-access-k4dnj\") pod \"fe898f81-5983-4e95-8807-ae70cf775321\" (UID: \"fe898f81-5983-4e95-8807-ae70cf775321\") " Sep 10 13:57:46.221482 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.221452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe898f81-5983-4e95-8807-ae70cf775321-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe898f81-5983-4e95-8807-ae70cf775321" (UID: "fe898f81-5983-4e95-8807-ae70cf775321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:57:46.223276 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.223243 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe898f81-5983-4e95-8807-ae70cf775321-kube-api-access-k4dnj" (OuterVolumeSpecName: "kube-api-access-k4dnj") pod "fe898f81-5983-4e95-8807-ae70cf775321" (UID: "fe898f81-5983-4e95-8807-ae70cf775321"). InnerVolumeSpecName "kube-api-access-k4dnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:46.322930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.322831 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe898f81-5983-4e95-8807-ae70cf775321-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:57:46.322930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.322864 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dnj\" (UniqueName: \"kubernetes.io/projected/fe898f81-5983-4e95-8807-ae70cf775321-kube-api-access-k4dnj\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:57:46.426044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.426011 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ccd18c-453a-4223-826f-fbfffb0b3472" path="/var/lib/kubelet/pods/08ccd18c-453a-4223-826f-fbfffb0b3472/volumes" Sep 10 13:57:46.449945 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.449906 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv"] Sep 10 13:57:46.455100 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:46.455067 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab74b3-58649fc6cc-blgqv"] Sep 10 13:57:48.424732 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:48.424692 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe898f81-5983-4e95-8807-ae70cf775321" path="/var/lib/kubelet/pods/fe898f81-5983-4e95-8807-ae70cf775321/volumes" Sep 10 13:57:52.459930 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:52.459884 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8"] Sep 10 13:57:54.713290 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:54.713256 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t"] Sep 10 13:57:54.713705 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:54.713460 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" containerID="cri-o://ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a" gracePeriod=30 Sep 10 13:57:55.146553 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.146510 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx"] Sep 10 13:57:55.146987 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.146965 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe898f81-5983-4e95-8807-ae70cf775321" containerName="payload-processing" Sep 10 13:57:55.147051 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.146989 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe898f81-5983-4e95-8807-ae70cf775321" containerName="payload-processing" Sep 10 13:57:55.147051 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.147002 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ccd18c-453a-4223-826f-fbfffb0b3472" containerName="revoke-keys" Sep 10 13:57:55.147051 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.147008 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ccd18c-453a-4223-826f-fbfffb0b3472" containerName="revoke-keys" Sep 10 13:57:55.147146 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.147075 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ccd18c-453a-4223-826f-fbfffb0b3472" containerName="revoke-keys" Sep 10 13:57:55.147146 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.147085 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe898f81-5983-4e95-8807-ae70cf775321" containerName="payload-processing" Sep 10 13:57:55.151301 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.151281 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:55.162944 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.162918 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:57:55.164281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.164257 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:57:55.171938 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.171910 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx"] Sep 10 13:57:55.209060 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.209024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895db669-3330-4958-9c4f-8621437d71b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx\" (UID: \"895db669-3330-4958-9c4f-8621437d71b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:55.309792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.309758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895db669-3330-4958-9c4f-8621437d71b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx\" (UID: \"895db669-3330-4958-9c4f-8621437d71b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:55.310454 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.310431 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895db669-3330-4958-9c4f-8621437d71b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx\" (UID: \"895db669-3330-4958-9c4f-8621437d71b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:55.461052 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.460965 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:55.736667 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:55.736641 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx"] Sep 10 13:57:55.738086 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:57:55.738062 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895db669_3330_4958_9c4f_8621437d71b5.slice/crio-e458e90bb38cd0e1b5c76733aac0d4da1b40fe45ba919286ad56e972e9928608 WatchSource:0}: Error finding container e458e90bb38cd0e1b5c76733aac0d4da1b40fe45ba919286ad56e972e9928608: Status 404 returned error can't find the container with id e458e90bb38cd0e1b5c76733aac0d4da1b40fe45ba919286ad56e972e9928608 Sep 10 13:57:56.178527 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:56.178492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" event={"ID":"895db669-3330-4958-9c4f-8621437d71b5","Type":"ContainerStarted","Data":"b62db0c1b998f57f07f1522ade08981fbab7b9a3c86a2f0be90694672331cdcc"} Sep 10 13:57:56.178769 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:56.178549 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" event={"ID":"895db669-3330-4958-9c4f-8621437d71b5","Type":"ContainerStarted","Data":"e458e90bb38cd0e1b5c76733aac0d4da1b40fe45ba919286ad56e972e9928608"} Sep 10 13:57:56.198905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:56.198805 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" podStartSLOduration=1.19878477 podStartE2EDuration="1.19878477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:57:56.198585898 +0000 UTC m=+1754.267585891" watchObservedRunningTime="2026-09-10 13:57:56.19878477 +0000 UTC m=+1754.267784739" Sep 10 13:57:57.186179 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:57.186147 2568 generic.go:356] "Generic (PLEG): container finished" podID="895db669-3330-4958-9c4f-8621437d71b5" containerID="b62db0c1b998f57f07f1522ade08981fbab7b9a3c86a2f0be90694672331cdcc" exitCode=0 Sep 10 13:57:57.186633 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:57.186218 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" event={"ID":"895db669-3330-4958-9c4f-8621437d71b5","Type":"ContainerDied","Data":"b62db0c1b998f57f07f1522ade08981fbab7b9a3c86a2f0be90694672331cdcc"} Sep 10 13:57:58.335697 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:58.335668 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:58.439467 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:58.439433 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895db669-3330-4958-9c4f-8621437d71b5-maas-api-service-ca\") pod \"895db669-3330-4958-9c4f-8621437d71b5\" (UID: \"895db669-3330-4958-9c4f-8621437d71b5\") " Sep 10 13:57:58.439815 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:58.439790 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895db669-3330-4958-9c4f-8621437d71b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "895db669-3330-4958-9c4f-8621437d71b5" (UID: "895db669-3330-4958-9c4f-8621437d71b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:57:58.540886 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:58.540840 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895db669-3330-4958-9c4f-8621437d71b5-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:57:59.199356 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:59.199329 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" Sep 10 13:57:59.199549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:59.199360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx" event={"ID":"895db669-3330-4958-9c4f-8621437d71b5","Type":"ContainerDied","Data":"e458e90bb38cd0e1b5c76733aac0d4da1b40fe45ba919286ad56e972e9928608"} Sep 10 13:57:59.199549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:59.199406 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e458e90bb38cd0e1b5c76733aac0d4da1b40fe45ba919286ad56e972e9928608" Sep 10 13:57:59.964627 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:57:59.964587 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:58:00.055720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055592 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdms\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-kube-api-access-7qdms\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.055720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055674 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5629076-33c1-4042-883a-4b1e4e458fff-istio-podinfo\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.055720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055698 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-credential-socket\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.055720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055717 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-istio-token\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.056044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055742 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5629076-33c1-4042-883a-4b1e4e458fff-istiod-ca-cert\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.056044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055798 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-certs\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.056044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055855 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-envoy\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.056044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055914 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-data\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.056044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.055951 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-socket\") pod \"e5629076-33c1-4042-883a-4b1e4e458fff\" (UID: \"e5629076-33c1-4042-883a-4b1e4e458fff\") " Sep 10 13:58:00.056044 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056021 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:00.056337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056107 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:00.056337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056179 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:00.056337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056323 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.056337 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056295 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:00.056547 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056342 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.056547 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056356 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.056547 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.056456 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5629076-33c1-4042-883a-4b1e4e458fff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:00.057990 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.057960 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5629076-33c1-4042-883a-4b1e4e458fff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:58:00.058096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.058020 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-kube-api-access-7qdms" (OuterVolumeSpecName: "kube-api-access-7qdms") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "kube-api-access-7qdms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:00.058278 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.058259 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:00.058313 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.058259 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5629076-33c1-4042-883a-4b1e4e458fff" (UID: "e5629076-33c1-4042-883a-4b1e4e458fff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:00.157873 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.157829 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.157873 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.157862 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5629076-33c1-4042-883a-4b1e4e458fff-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.157873 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.157872 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdms\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-kube-api-access-7qdms\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.157873 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.157884 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5629076-33c1-4042-883a-4b1e4e458fff-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.158154 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.157892 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5629076-33c1-4042-883a-4b1e4e458fff-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.158154 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.157901 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5629076-33c1-4042-883a-4b1e4e458fff-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:00.207427 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.207386 2568 generic.go:356] "Generic (PLEG): container finished" podID="e5629076-33c1-4042-883a-4b1e4e458fff" containerID="ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a" exitCode=0 Sep 10 13:58:00.207716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.207440 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" Sep 10 13:58:00.207716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.207474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" event={"ID":"e5629076-33c1-4042-883a-4b1e4e458fff","Type":"ContainerDied","Data":"ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a"} Sep 10 13:58:00.207716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.207533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t" event={"ID":"e5629076-33c1-4042-883a-4b1e4e458fff","Type":"ContainerDied","Data":"06f64764adaafef1805a7ba083218400a8bc6170ed4e6dd481272ada57c97bf4"} Sep 10 13:58:00.207716 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.207564 2568 scope.go:117] "RemoveContainer" containerID="ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a" Sep 10 13:58:00.217751 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.217728 2568 scope.go:117] "RemoveContainer" containerID="de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195" Sep 10 13:58:00.229275 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.229037 2568 scope.go:117] "RemoveContainer" containerID="ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a" Sep 10 13:58:00.229487 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:58:00.229462 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a\": container with ID starting with ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a not found: ID does not exist" containerID="ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a" Sep 10 13:58:00.229550 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.229526 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a"} err="failed to get container status \"ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a\": rpc error: code = NotFound desc = could not find container \"ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a\": container with ID starting with ecbb5c71eb74a5f8e924031495b0651972b801d3e00583d2c3372822c9f26a8a not found: ID does not exist" Sep 10 13:58:00.229637 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.229555 2568 scope.go:117] "RemoveContainer" containerID="de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195" Sep 10 13:58:00.229939 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:58:00.229919 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195\": container with ID starting with de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195 not found: ID does not exist" containerID="de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195" Sep 10 13:58:00.230027 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.229947 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195"} err="failed to get container status \"de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195\": rpc error: code = NotFound desc = could not find container \"de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195\": container with ID starting with de11abef399cbeabc5113b78296794eca0625047f990851de4fb57370d577195 not found: ID does not exist" Sep 10 13:58:00.242894 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.242860 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t"] Sep 10 13:58:00.249824 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.249794 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ab74b3-openshift-default-56fc6f97db-99s7t"] Sep 10 13:58:00.424433 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:00.424349 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" path="/var/lib/kubelet/pods/e5629076-33c1-4042-883a-4b1e4e458fff/volumes" Sep 10 13:58:07.480266 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.480201 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" podUID="e8a1b93a-76aa-4258-9aea-bcc4edd674dc" containerName="main" containerID="cri-o://7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a" gracePeriod=45 Sep 10 13:58:07.741009 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.740939 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:58:07.837720 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.837681 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tmp-dir\") pod \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " Sep 10 13:58:07.837904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.837756 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tls-certs\") pod \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " Sep 10 13:58:07.837904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.837778 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfpfs\" (UniqueName: \"kubernetes.io/projected/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-kube-api-access-jfpfs\") pod \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " Sep 10 13:58:07.837904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.837820 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-dshm\") pod \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " Sep 10 13:58:07.837904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.837843 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-model-cache\") pod \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " Sep 10 13:58:07.837904 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.837906 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-home\") pod \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\" (UID: \"e8a1b93a-76aa-4258-9aea-bcc4edd674dc\") " Sep 10 13:58:07.838175 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.838010 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8a1b93a-76aa-4258-9aea-bcc4edd674dc" (UID: "e8a1b93a-76aa-4258-9aea-bcc4edd674dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:07.838229 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.838188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8a1b93a-76aa-4258-9aea-bcc4edd674dc" (UID: "e8a1b93a-76aa-4258-9aea-bcc4edd674dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:07.838229 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.838219 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tmp-dir\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:07.838308 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.838225 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-home" (OuterVolumeSpecName: "home") pod "e8a1b93a-76aa-4258-9aea-bcc4edd674dc" (UID: "e8a1b93a-76aa-4258-9aea-bcc4edd674dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:07.840144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.840115 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-dshm" (OuterVolumeSpecName: "dshm") pod "e8a1b93a-76aa-4258-9aea-bcc4edd674dc" (UID: "e8a1b93a-76aa-4258-9aea-bcc4edd674dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:07.840262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.840147 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-kube-api-access-jfpfs" (OuterVolumeSpecName: "kube-api-access-jfpfs") pod "e8a1b93a-76aa-4258-9aea-bcc4edd674dc" (UID: "e8a1b93a-76aa-4258-9aea-bcc4edd674dc"). InnerVolumeSpecName "kube-api-access-jfpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:07.840262 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.840169 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8a1b93a-76aa-4258-9aea-bcc4edd674dc" (UID: "e8a1b93a-76aa-4258-9aea-bcc4edd674dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:58:07.939243 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.939206 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-model-cache\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:07.939243 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.939237 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-home\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:07.939243 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.939246 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-tls-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:07.939243 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.939255 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfpfs\" (UniqueName: \"kubernetes.io/projected/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-kube-api-access-jfpfs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:07.939528 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:07.939263 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a1b93a-76aa-4258-9aea-bcc4edd674dc-dshm\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:08.259988 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.259949 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a1b93a-76aa-4258-9aea-bcc4edd674dc" containerID="7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a" exitCode=0 Sep 10 13:58:08.260191 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.259994 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" Sep 10 13:58:08.260191 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.260032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" event={"ID":"e8a1b93a-76aa-4258-9aea-bcc4edd674dc","Type":"ContainerDied","Data":"7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a"} Sep 10 13:58:08.260191 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.260079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8" event={"ID":"e8a1b93a-76aa-4258-9aea-bcc4edd674dc","Type":"ContainerDied","Data":"f85d87b5525aff5e5c84bb11cd81ff2b4917709856ced44d3b586d66a1ceaeef"} Sep 10 13:58:08.260191 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.260098 2568 scope.go:117] "RemoveContainer" containerID="7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a" Sep 10 13:58:08.269883 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.269862 2568 scope.go:117] "RemoveContainer" containerID="7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a" Sep 10 13:58:08.270171 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:58:08.270152 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a\": container with ID starting with 7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a not found: ID does not exist" containerID="7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a" Sep 10 13:58:08.270219 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.270180 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a"} err="failed to get container status \"7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a\": rpc error: code = NotFound desc = could not find container \"7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a\": container with ID starting with 7af39a5d11ac1ca01070500d283854678fbe88e26b6bc202cca7d80039e5a13a not found: ID does not exist" Sep 10 13:58:08.287706 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.287662 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8"] Sep 10 13:58:08.298006 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.297967 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/sub-test-model-7a11e9-kserve-75f6dd4787-cmmw8"] Sep 10 13:58:08.424879 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:08.424839 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a1b93a-76aa-4258-9aea-bcc4edd674dc" path="/var/lib/kubelet/pods/e8a1b93a-76aa-4258-9aea-bcc4edd674dc/volumes" Sep 10 13:58:11.048252 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.048212 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22"] Sep 10 13:58:11.048671 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.048493 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" podUID="5b128fc9-ac3a-4426-abc4-76455e328f65" containerName="payload-processing" containerID="cri-o://66d0ed5300240496f7a430071282451b5e832a3cd9dc14a23642ac03597166eb" gracePeriod=30 Sep 10 13:58:11.280275 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.280240 2568 generic.go:356] "Generic (PLEG): container finished" podID="5b128fc9-ac3a-4426-abc4-76455e328f65" containerID="66d0ed5300240496f7a430071282451b5e832a3cd9dc14a23642ac03597166eb" exitCode=0 Sep 10 13:58:11.280462 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.280281 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" event={"ID":"5b128fc9-ac3a-4426-abc4-76455e328f65","Type":"ContainerDied","Data":"66d0ed5300240496f7a430071282451b5e832a3cd9dc14a23642ac03597166eb"} Sep 10 13:58:11.280544 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.280525 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx"] Sep 10 13:58:11.285993 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.285957 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8aa282-rqdmx"] Sep 10 13:58:11.312457 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.312382 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:58:11.371858 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.371816 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b128fc9-ac3a-4426-abc4-76455e328f65-plugins-config-volume\") pod \"5b128fc9-ac3a-4426-abc4-76455e328f65\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " Sep 10 13:58:11.372059 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.371992 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmjpd\" (UniqueName: \"kubernetes.io/projected/5b128fc9-ac3a-4426-abc4-76455e328f65-kube-api-access-gmjpd\") pod \"5b128fc9-ac3a-4426-abc4-76455e328f65\" (UID: \"5b128fc9-ac3a-4426-abc4-76455e328f65\") " Sep 10 13:58:11.372646 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.372577 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b128fc9-ac3a-4426-abc4-76455e328f65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b128fc9-ac3a-4426-abc4-76455e328f65" (UID: "5b128fc9-ac3a-4426-abc4-76455e328f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:11.375304 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.375267 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b128fc9-ac3a-4426-abc4-76455e328f65-kube-api-access-gmjpd" (OuterVolumeSpecName: "kube-api-access-gmjpd") pod "5b128fc9-ac3a-4426-abc4-76455e328f65" (UID: "5b128fc9-ac3a-4426-abc4-76455e328f65"). InnerVolumeSpecName "kube-api-access-gmjpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:11.473400 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.473355 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjpd\" (UniqueName: \"kubernetes.io/projected/5b128fc9-ac3a-4426-abc4-76455e328f65-kube-api-access-gmjpd\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:11.473400 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:11.473390 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b128fc9-ac3a-4426-abc4-76455e328f65-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.287488 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.287460 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" Sep 10 13:58:12.287974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.287459 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22" event={"ID":"5b128fc9-ac3a-4426-abc4-76455e328f65","Type":"ContainerDied","Data":"377f3a1d820915f77e089f70711042d721c3f66dce476503ffee74e515acd540"} Sep 10 13:58:12.287974 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.287632 2568 scope.go:117] "RemoveContainer" containerID="66d0ed5300240496f7a430071282451b5e832a3cd9dc14a23642ac03597166eb" Sep 10 13:58:12.330798 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.330758 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22"] Sep 10 13:58:12.340735 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.340706 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8aa282-5cfcc856b9-t8r22"] Sep 10 13:58:12.424344 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.424310 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b128fc9-ac3a-4426-abc4-76455e328f65" path="/var/lib/kubelet/pods/5b128fc9-ac3a-4426-abc4-76455e328f65/volumes" Sep 10 13:58:12.424704 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:12.424690 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895db669-3330-4958-9c4f-8621437d71b5" path="/var/lib/kubelet/pods/895db669-3330-4958-9c4f-8621437d71b5/volumes" Sep 10 13:58:42.489385 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:42.489353 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:58:42.501760 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:42.501729 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:58:42.503127 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:42.503098 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 13:58:42.510424 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:42.510403 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 13:58:58.164643 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:58.164591 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs"] Sep 10 13:58:58.165148 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:58:58.164828 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" containerID="cri-o://74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7" gracePeriod=30 Sep 10 13:59:03.419328 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.419297 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:59:03.505312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505267 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-envoy\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505312 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505310 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-socket\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505353 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5346b571-363d-4a03-921b-33ee7b2f8501-istiod-ca-cert\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5346b571-363d-4a03-921b-33ee7b2f8501-istio-podinfo\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505413 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkjr8\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-kube-api-access-lkjr8\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505433 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-istio-token\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505467 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-certs\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505492 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-credential-socket\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.505630 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505542 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-data\") pod \"5346b571-363d-4a03-921b-33ee7b2f8501\" (UID: \"5346b571-363d-4a03-921b-33ee7b2f8501\") " Sep 10 13:59:03.506005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505676 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:59:03.506005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505900 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:59:03.506005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505925 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:59:03.506005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.505943 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5346b571-363d-4a03-921b-33ee7b2f8501-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:59:03.506225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.506026 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.506225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.506045 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.506225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.506060 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.506225 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.506074 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5346b571-363d-4a03-921b-33ee7b2f8501-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.506405 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.506383 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-data" (OuterVolumeSpecName: "istio-data") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:59:03.507805 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.507765 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-kube-api-access-lkjr8" (OuterVolumeSpecName: "kube-api-access-lkjr8") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "kube-api-access-lkjr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:59:03.508094 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.508066 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5346b571-363d-4a03-921b-33ee7b2f8501-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:59:03.508094 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.508075 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:59:03.508228 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.508113 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-istio-token" (OuterVolumeSpecName: "istio-token") pod "5346b571-363d-4a03-921b-33ee7b2f8501" (UID: "5346b571-363d-4a03-921b-33ee7b2f8501"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:59:03.597322 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.597280 2568 generic.go:356] "Generic (PLEG): container finished" podID="5346b571-363d-4a03-921b-33ee7b2f8501" containerID="74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7" exitCode=0 Sep 10 13:59:03.597502 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.597338 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" Sep 10 13:59:03.597502 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.597365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" event={"ID":"5346b571-363d-4a03-921b-33ee7b2f8501","Type":"ContainerDied","Data":"74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7"} Sep 10 13:59:03.597502 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.597427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs" event={"ID":"5346b571-363d-4a03-921b-33ee7b2f8501","Type":"ContainerDied","Data":"be09707a3ba0ee23c256f3ce76d629f9c64fd177e37513f306928319098d760d"} Sep 10 13:59:03.597502 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.597453 2568 scope.go:117] "RemoveContainer" containerID="74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7" Sep 10 13:59:03.607299 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.607273 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.607299 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.607299 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5346b571-363d-4a03-921b-33ee7b2f8501-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.607452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.607308 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjr8\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-kube-api-access-lkjr8\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.607452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.607318 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5346b571-363d-4a03-921b-33ee7b2f8501-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.607452 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.607326 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5346b571-363d-4a03-921b-33ee7b2f8501-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 13:59:03.607881 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.607862 2568 scope.go:117] "RemoveContainer" containerID="2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675" Sep 10 13:59:03.617170 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.617151 2568 scope.go:117] "RemoveContainer" containerID="74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7" Sep 10 13:59:03.617476 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:59:03.617455 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7\": container with ID starting with 74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7 not found: ID does not exist" containerID="74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7" Sep 10 13:59:03.617549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.617486 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7"} err="failed to get container status \"74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7\": rpc error: code = NotFound desc = could not find container \"74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7\": container with ID starting with 74e30aa67450ce7644b9c7e2432340befe173f4b3d2bc24e4438830b435005a7 not found: ID does not exist" Sep 10 13:59:03.617549 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.617511 2568 scope.go:117] "RemoveContainer" containerID="2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675" Sep 10 13:59:03.617797 ip-10-0-139-172 kubenswrapper[2568]: E0910 13:59:03.617777 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675\": container with ID starting with 2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675 not found: ID does not exist" containerID="2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675" Sep 10 13:59:03.617842 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.617804 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675"} err="failed to get container status \"2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675\": rpc error: code = NotFound desc = could not find container \"2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675\": container with ID starting with 2380afe40edf925c098278ae4ebd920c4bb3078d1146dc7b7a4a9b216ed64675 not found: ID does not exist" Sep 10 13:59:03.627358 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.627328 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs"] Sep 10 13:59:03.632348 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:03.632319 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a310dd-openshift-default-669bd4fbbb-rf2gs"] Sep 10 13:59:04.423869 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:04.423831 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" path="/var/lib/kubelet/pods/5346b571-363d-4a03-921b-33ee7b2f8501/volumes" Sep 10 13:59:20.281821 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.281785 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8"] Sep 10 13:59:20.282358 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282341 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" Sep 10 13:59:20.282426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282362 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" Sep 10 13:59:20.282426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282389 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" Sep 10 13:59:20.282426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282397 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" Sep 10 13:59:20.282426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282410 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b128fc9-ac3a-4426-abc4-76455e328f65" containerName="payload-processing" Sep 10 13:59:20.282426 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282418 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b128fc9-ac3a-4426-abc4-76455e328f65" containerName="payload-processing" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282434 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282443 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282451 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a1b93a-76aa-4258-9aea-bcc4edd674dc" containerName="main" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282459 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a1b93a-76aa-4258-9aea-bcc4edd674dc" containerName="main" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282472 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895db669-3330-4958-9c4f-8621437d71b5" containerName="revoke-keys" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282479 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="895db669-3330-4958-9c4f-8621437d71b5" containerName="revoke-keys" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282559 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a1b93a-76aa-4258-9aea-bcc4edd674dc" containerName="main" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282569 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" Sep 10 13:59:20.282586 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282578 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" Sep 10 13:59:20.282896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282592 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5629076-33c1-4042-883a-4b1e4e458fff" containerName="istio-proxy" Sep 10 13:59:20.282896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282616 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b128fc9-ac3a-4426-abc4-76455e328f65" containerName="payload-processing" Sep 10 13:59:20.282896 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.282624 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="895db669-3330-4958-9c4f-8621437d71b5" containerName="revoke-keys" Sep 10 13:59:20.285773 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.285754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.288612 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.288571 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3f01c7-openshift-default-dockercfg-vp6vw\"" Sep 10 13:59:20.300494 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.300465 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8"] Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwz5\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-kube-api-access-swwz5\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-credential-socket\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-data\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-envoy\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47d032e-e443-49b2-bacf-755c5ad4511b-istiod-ca-cert\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-token\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.373456 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-podinfo\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.374037 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373492 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-certs\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.374037 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.373564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-socket\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474250 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-socket\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwz5\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-kube-api-access-swwz5\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-credential-socket\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474343 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-data\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-envoy\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474404 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47d032e-e443-49b2-bacf-755c5ad4511b-istiod-ca-cert\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-token\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474472 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-podinfo\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-certs\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474736 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-socket\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-credential-socket\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.474905 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.474858 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-certs\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.475062 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.475019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-data\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.475154 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.475136 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47d032e-e443-49b2-bacf-755c5ad4511b-istiod-ca-cert\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.477005 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.476983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-envoy\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.477143 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.477128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-podinfo\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.486695 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.486671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-token\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.486839 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.486727 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwz5\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-kube-api-access-swwz5\") pod \"e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.597878 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.597780 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:20.897229 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.897198 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8"] Sep 10 13:59:20.898481 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:59:20.898454 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47d032e_e443_49b2_bacf_755c5ad4511b.slice/crio-c7f80e366e46d03411e1e6ea87dbec4b29fecb1ad8e735f63634b58bbeb49844 WatchSource:0}: Error finding container c7f80e366e46d03411e1e6ea87dbec4b29fecb1ad8e735f63634b58bbeb49844: Status 404 returned error can't find the container with id c7f80e366e46d03411e1e6ea87dbec4b29fecb1ad8e735f63634b58bbeb49844 Sep 10 13:59:20.900691 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.900660 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:59:20.900754 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.900731 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:59:20.900792 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:20.900765 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:59:21.705996 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:21.705955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" event={"ID":"b47d032e-e443-49b2-bacf-755c5ad4511b","Type":"ContainerStarted","Data":"8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596"} Sep 10 13:59:21.706405 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:21.706051 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" event={"ID":"b47d032e-e443-49b2-bacf-755c5ad4511b","Type":"ContainerStarted","Data":"c7f80e366e46d03411e1e6ea87dbec4b29fecb1ad8e735f63634b58bbeb49844"} Sep 10 13:59:22.598875 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:22.598838 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:22.604230 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:22.604205 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:22.627736 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:22.627680 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podStartSLOduration=2.627662132 podStartE2EDuration="2.627662132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:21.736629143 +0000 UTC m=+1839.805629099" watchObservedRunningTime="2026-09-10 13:59:22.627662132 +0000 UTC m=+1840.696662097" Sep 10 13:59:22.711892 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:22.711861 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:22.712870 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:22.712847 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:38.753788 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:38.753740 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:59:38.754173 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:38.753836 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:59:39.810589 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:39.810557 2568 generic.go:356] "Generic (PLEG): container finished" podID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerID="8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596" exitCode=0 Sep 10 13:59:39.811039 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:39.810633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" event={"ID":"b47d032e-e443-49b2-bacf-755c5ad4511b","Type":"ContainerDied","Data":"8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596"} Sep 10 13:59:39.811135 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:39.811117 2568 scope.go:117] "RemoveContainer" containerID="8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596" Sep 10 13:59:39.812271 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:39.812236 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:59:39.812326 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:39.812314 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:59:39.812410 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:39.812367 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 13:59:40.818294 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:40.818251 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" event={"ID":"b47d032e-e443-49b2-bacf-755c5ad4511b","Type":"ContainerStarted","Data":"89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903"} Sep 10 13:59:41.598281 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:41.598247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:41.598483 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:41.598454 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:59:41.598542 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:41.598517 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 10 13:59:42.598477 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:42.598431 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:59:42.598943 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:42.598516 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 10 13:59:43.599144 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:43.599103 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:59:43.599545 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:43.599164 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 10 13:59:44.599196 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:44.599149 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:59:44.599735 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:44.599220 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 10 13:59:45.598770 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:45.598729 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:59:45.598957 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:45.598791 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 10 13:59:46.601888 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:46.601856 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:46.602362 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:46.602140 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:46.602799 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:46.602782 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 13:59:55.154516 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.154479 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm"] Sep 10 13:59:55.154967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.154919 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" Sep 10 13:59:55.154967 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.154931 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" Sep 10 13:59:55.155049 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.154985 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5346b571-363d-4a03-921b-33ee7b2f8501" containerName="istio-proxy" Sep 10 13:59:55.158285 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.158268 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.163096 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.163072 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0741a\"" Sep 10 13:59:55.170620 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.170570 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm"] Sep 10 13:59:55.220797 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.220761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvgv\" (UniqueName: \"kubernetes.io/projected/71cc6a20-0222-42c2-a6f5-a5258135e652-kube-api-access-4nvgv\") pod \"payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.220980 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.220836 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cc6a20-0222-42c2-a6f5-a5258135e652-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.322163 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.322118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvgv\" (UniqueName: \"kubernetes.io/projected/71cc6a20-0222-42c2-a6f5-a5258135e652-kube-api-access-4nvgv\") pod \"payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.322360 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.322240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cc6a20-0222-42c2-a6f5-a5258135e652-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.323016 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.322983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cc6a20-0222-42c2-a6f5-a5258135e652-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.334189 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.334160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvgv\" (UniqueName: \"kubernetes.io/projected/71cc6a20-0222-42c2-a6f5-a5258135e652-kube-api-access-4nvgv\") pod \"payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.468998 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.468957 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.758998 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.758969 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm"] Sep 10 13:59:55.760936 ip-10-0-139-172 kubenswrapper[2568]: W0910 13:59:55.760894 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cc6a20_0222_42c2_a6f5_a5258135e652.slice/crio-da8e473fbad2f04238e999e48210dd69c42709ef7ea3650aef35996be3f704d9 WatchSource:0}: Error finding container da8e473fbad2f04238e999e48210dd69c42709ef7ea3650aef35996be3f704d9: Status 404 returned error can't find the container with id da8e473fbad2f04238e999e48210dd69c42709ef7ea3650aef35996be3f704d9 Sep 10 13:59:55.924376 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.924335 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" event={"ID":"71cc6a20-0222-42c2-a6f5-a5258135e652","Type":"ContainerStarted","Data":"386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d"} Sep 10 13:59:55.924570 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.924402 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" event={"ID":"71cc6a20-0222-42c2-a6f5-a5258135e652","Type":"ContainerStarted","Data":"da8e473fbad2f04238e999e48210dd69c42709ef7ea3650aef35996be3f704d9"} Sep 10 13:59:55.924570 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.924529 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 13:59:55.945907 ip-10-0-139-172 kubenswrapper[2568]: I0910 13:59:55.945848 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" podStartSLOduration=0.945826505 podStartE2EDuration="945.826505ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:55.943865861 +0000 UTC m=+1874.012865827" watchObservedRunningTime="2026-09-10 13:59:55.945826505 +0000 UTC m=+1874.014826474" Sep 10 14:00:00.143285 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.143245 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m"] Sep 10 14:00:00.146993 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.146972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" Sep 10 14:00:00.148441 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.148418 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7"] Sep 10 14:00:00.149586 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.149565 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:00:00.151715 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.151693 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" Sep 10 14:00:00.151832 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.151795 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx"] Sep 10 14:00:00.155388 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.155369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" Sep 10 14:00:00.158899 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.158880 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m"] Sep 10 14:00:00.160117 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.159727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" Sep 10 14:00:00.167591 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.167570 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx"] Sep 10 14:00:00.171904 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.171492 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7"] Sep 10 14:00:00.172840 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.172823 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" Sep 10 14:00:00.215958 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.215913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" Sep 10 14:00:00.462900 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.462874 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m"] Sep 10 14:00:00.465633 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:00:00.465592 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490a0ef1_265c_40d2_b458_a430ab2f86c9.slice/crio-eb0831e27521f46395453ec109e5ee05166610e65311e1934805a80d6eedb8fe WatchSource:0}: Error finding container eb0831e27521f46395453ec109e5ee05166610e65311e1934805a80d6eedb8fe: Status 404 returned error can't find the container with id eb0831e27521f46395453ec109e5ee05166610e65311e1934805a80d6eedb8fe Sep 10 14:00:00.470019 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.469995 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7"] Sep 10 14:00:00.471325 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:00:00.471293 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0292118_7b75_4d46_bc1b_429d2fb76b5b.slice/crio-5059f4cba6fcb60e1c472bb19146063e13a504c58dbbc91805bd3c3e9d0f5569 WatchSource:0}: Error finding container 5059f4cba6fcb60e1c472bb19146063e13a504c58dbbc91805bd3c3e9d0f5569: Status 404 returned error can't find the container with id 5059f4cba6fcb60e1c472bb19146063e13a504c58dbbc91805bd3c3e9d0f5569 Sep 10 14:00:00.511545 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.511524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx"] Sep 10 14:00:00.513297 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:00:00.513271 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1fcd5d_dc70_42c6_a56a_b64af934e26b.slice/crio-6eb3b72035386acf4c7aff4083fde983c2c899b78010b6a105bfc66583fcfe2b WatchSource:0}: Error finding container 6eb3b72035386acf4c7aff4083fde983c2c899b78010b6a105bfc66583fcfe2b: Status 404 returned error can't find the container with id 6eb3b72035386acf4c7aff4083fde983c2c899b78010b6a105bfc66583fcfe2b Sep 10 14:00:00.952546 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.952510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" event={"ID":"ba1fcd5d-dc70-42c6-a56a-b64af934e26b","Type":"ContainerStarted","Data":"11966c6e1f289e09bf3180e97a6aaca5cc9a208013fde54edb67e665761312b9"} Sep 10 14:00:00.952775 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.952573 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" event={"ID":"ba1fcd5d-dc70-42c6-a56a-b64af934e26b","Type":"ContainerStarted","Data":"6eb3b72035386acf4c7aff4083fde983c2c899b78010b6a105bfc66583fcfe2b"} Sep 10 14:00:00.954043 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.954015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" event={"ID":"490a0ef1-265c-40d2-b458-a430ab2f86c9","Type":"ContainerStarted","Data":"66241bb6091c7292d04576fd320705645b6b3ebcddc7ea5d5f8fad39fd446d24"} Sep 10 14:00:00.954147 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.954062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" event={"ID":"490a0ef1-265c-40d2-b458-a430ab2f86c9","Type":"ContainerStarted","Data":"eb0831e27521f46395453ec109e5ee05166610e65311e1934805a80d6eedb8fe"} Sep 10 14:00:00.955429 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.955405 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" event={"ID":"e0292118-7b75-4d46-bc1b-429d2fb76b5b","Type":"ContainerStarted","Data":"d209f0ba30d351b6e6295b1b6d7a8ad582ff5dac4f4740edc651ebcdba721481"} Sep 10 14:00:00.955533 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.955445 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" event={"ID":"e0292118-7b75-4d46-bc1b-429d2fb76b5b","Type":"ContainerStarted","Data":"5059f4cba6fcb60e1c472bb19146063e13a504c58dbbc91805bd3c3e9d0f5569"} Sep 10 14:00:00.972431 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.972382 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" podStartSLOduration=0.972368402 podStartE2EDuration="972.368402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:00.971986488 +0000 UTC m=+1879.040986454" watchObservedRunningTime="2026-09-10 14:00:00.972368402 +0000 UTC m=+1879.041368400" Sep 10 14:00:00.991361 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:00.991261 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" podStartSLOduration=0.991244082 podStartE2EDuration="991.244082ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:00.989522821 +0000 UTC m=+1879.058522783" watchObservedRunningTime="2026-09-10 14:00:00.991244082 +0000 UTC m=+1879.060244047" Sep 10 14:00:01.010762 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.010706 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" podStartSLOduration=1.010692167 podStartE2EDuration="1.010692167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:01.008513483 +0000 UTC m=+1879.077513448" watchObservedRunningTime="2026-09-10 14:00:01.010692167 +0000 UTC m=+1879.079692132" Sep 10 14:00:01.962316 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.962278 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba1fcd5d-dc70-42c6-a56a-b64af934e26b" containerID="11966c6e1f289e09bf3180e97a6aaca5cc9a208013fde54edb67e665761312b9" exitCode=0 Sep 10 14:00:01.962781 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.962364 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" event={"ID":"ba1fcd5d-dc70-42c6-a56a-b64af934e26b","Type":"ContainerDied","Data":"11966c6e1f289e09bf3180e97a6aaca5cc9a208013fde54edb67e665761312b9"} Sep 10 14:00:01.963931 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.963909 2568 generic.go:356] "Generic (PLEG): container finished" podID="490a0ef1-265c-40d2-b458-a430ab2f86c9" containerID="66241bb6091c7292d04576fd320705645b6b3ebcddc7ea5d5f8fad39fd446d24" exitCode=0 Sep 10 14:00:01.964027 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.963993 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" event={"ID":"490a0ef1-265c-40d2-b458-a430ab2f86c9","Type":"ContainerDied","Data":"66241bb6091c7292d04576fd320705645b6b3ebcddc7ea5d5f8fad39fd446d24"} Sep 10 14:00:01.965486 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.965468 2568 generic.go:356] "Generic (PLEG): container finished" podID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerID="d209f0ba30d351b6e6295b1b6d7a8ad582ff5dac4f4740edc651ebcdba721481" exitCode=7 Sep 10 14:00:01.965573 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.965506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" event={"ID":"e0292118-7b75-4d46-bc1b-429d2fb76b5b","Type":"ContainerDied","Data":"d209f0ba30d351b6e6295b1b6d7a8ad582ff5dac4f4740edc651ebcdba721481"} Sep 10 14:00:01.965727 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:01.965712 2568 scope.go:117] "RemoveContainer" containerID="d209f0ba30d351b6e6295b1b6d7a8ad582ff5dac4f4740edc651ebcdba721481" Sep 10 14:00:02.973750 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:02.973719 2568 generic.go:356] "Generic (PLEG): container finished" podID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerID="cc0b54b03f0abfab206febc3f82817e96d0e33fe78095f3de48a4efa430d4474" exitCode=0 Sep 10 14:00:02.974106 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:02.973803 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" event={"ID":"e0292118-7b75-4d46-bc1b-429d2fb76b5b","Type":"ContainerDied","Data":"cc0b54b03f0abfab206febc3f82817e96d0e33fe78095f3de48a4efa430d4474"} Sep 10 14:00:02.974106 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:02.973866 2568 scope.go:117] "RemoveContainer" containerID="d209f0ba30d351b6e6295b1b6d7a8ad582ff5dac4f4740edc651ebcdba721481" Sep 10 14:00:03.120944 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.120920 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" Sep 10 14:00:03.157357 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.157329 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" Sep 10 14:00:03.980871 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.980838 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" Sep 10 14:00:03.981327 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.980875 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m" event={"ID":"490a0ef1-265c-40d2-b458-a430ab2f86c9","Type":"ContainerDied","Data":"eb0831e27521f46395453ec109e5ee05166610e65311e1934805a80d6eedb8fe"} Sep 10 14:00:03.981327 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.980924 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0831e27521f46395453ec109e5ee05166610e65311e1934805a80d6eedb8fe" Sep 10 14:00:03.984091 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.984062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" event={"ID":"ba1fcd5d-dc70-42c6-a56a-b64af934e26b","Type":"ContainerDied","Data":"6eb3b72035386acf4c7aff4083fde983c2c899b78010b6a105bfc66583fcfe2b"} Sep 10 14:00:03.984091 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.984082 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx" Sep 10 14:00:03.984249 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:03.984116 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb3b72035386acf4c7aff4083fde983c2c899b78010b6a105bfc66583fcfe2b" Sep 10 14:00:04.121639 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:04.121609 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" Sep 10 14:00:04.991524 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:04.991450 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" Sep 10 14:00:04.991961 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:04.991449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7" event={"ID":"e0292118-7b75-4d46-bc1b-429d2fb76b5b","Type":"ContainerDied","Data":"5059f4cba6fcb60e1c472bb19146063e13a504c58dbbc91805bd3c3e9d0f5569"} Sep 10 14:00:04.991961 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:04.991578 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5059f4cba6fcb60e1c472bb19146063e13a504c58dbbc91805bd3c3e9d0f5569" Sep 10 14:00:06.931709 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:06.931677 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 14:00:22.836712 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:22.836682 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mbq7f_e1be089e-0c86-496e-82b2-592718142acb/payload-processing/0.log" Sep 10 14:00:22.968986 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:22.968953 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm_71cc6a20-0222-42c2-a6f5-a5258135e652/payload-processing/0.log" Sep 10 14:00:42.711047 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:42.711009 2568 scope.go:117] "RemoveContainer" containerID="22dc109e4d696229b6784ae0afe83ee08ec6f16d100f3919b817c80a9b03dd95" Sep 10 14:00:53.730703 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:53.730661 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mbq7f_e1be089e-0c86-496e-82b2-592718142acb/payload-processing/0.log" Sep 10 14:00:53.874159 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:53.874134 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm_71cc6a20-0222-42c2-a6f5-a5258135e652/payload-processing/0.log" Sep 10 14:00:54.510755 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.510724 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb"] Sep 10 14:00:54.511166 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511154 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1fcd5d-dc70-42c6-a56a-b64af934e26b" containerName="cleanup" Sep 10 14:00:54.511218 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511168 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1fcd5d-dc70-42c6-a56a-b64af934e26b" containerName="cleanup" Sep 10 14:00:54.511218 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511182 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerName="cleanup" Sep 10 14:00:54.511218 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511188 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerName="cleanup" Sep 10 14:00:54.511218 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511211 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490a0ef1-265c-40d2-b458-a430ab2f86c9" containerName="cleanup" Sep 10 14:00:54.511346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511227 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="490a0ef1-265c-40d2-b458-a430ab2f86c9" containerName="cleanup" Sep 10 14:00:54.511346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511283 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerName="cleanup" Sep 10 14:00:54.511346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511293 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerName="cleanup" Sep 10 14:00:54.511346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511299 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="490a0ef1-265c-40d2-b458-a430ab2f86c9" containerName="cleanup" Sep 10 14:00:54.511346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.511306 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1fcd5d-dc70-42c6-a56a-b64af934e26b" containerName="cleanup" Sep 10 14:00:54.514354 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.514335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.517726 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.517707 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-eb3d6d-openshift-default-dockercfg-7n24r\"" Sep 10 14:00:54.535970 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.535941 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb"] Sep 10 14:00:54.595173 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f7a48-3803-4456-97ac-f6a9c78069db-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595379 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f7a48-3803-4456-97ac-f6a9c78069db-istio-podinfo\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595379 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-envoy\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595379 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595323 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-certs\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595554 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595378 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-data\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595554 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884j7\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-kube-api-access-884j7\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595554 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-credential-socket\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595554 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-istio-token\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.595820 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.595590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-socket\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696193 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-socket\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696407 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f7a48-3803-4456-97ac-f6a9c78069db-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696407 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f7a48-3803-4456-97ac-f6a9c78069db-istio-podinfo\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696407 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-envoy\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696407 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-certs\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696407 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-data\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884j7\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-kube-api-access-884j7\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-credential-socket\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-istio-token\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-socket\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.696728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696695 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-certs\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.697069 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.696941 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-data\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.697069 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.697055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-credential-socket\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.697161 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.697065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f7a48-3803-4456-97ac-f6a9c78069db-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.698710 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.698687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-envoy\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.699154 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.699131 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f7a48-3803-4456-97ac-f6a9c78069db-istio-podinfo\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.706210 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.706184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-istio-token\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.706300 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.706184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-884j7\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-kube-api-access-884j7\") pod \"e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:54.826839 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:54.826750 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:55.113501 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.113475 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb"] Sep 10 14:00:55.114848 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:00:55.114819 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000f7a48_3803_4456_97ac_f6a9c78069db.slice/crio-0d88ab1d84881223dee377d1508f2decc5e6318c1ec6b43c51abf6040f0228b9 WatchSource:0}: Error finding container 0d88ab1d84881223dee377d1508f2decc5e6318c1ec6b43c51abf6040f0228b9: Status 404 returned error can't find the container with id 0d88ab1d84881223dee377d1508f2decc5e6318c1ec6b43c51abf6040f0228b9 Sep 10 14:00:55.117172 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.117136 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:00:55.117281 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.117217 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:00:55.117346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.117285 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:00:55.274922 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.274886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" event={"ID":"000f7a48-3803-4456-97ac-f6a9c78069db","Type":"ContainerStarted","Data":"7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8"} Sep 10 14:00:55.275124 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.274956 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" event={"ID":"000f7a48-3803-4456-97ac-f6a9c78069db","Type":"ContainerStarted","Data":"0d88ab1d84881223dee377d1508f2decc5e6318c1ec6b43c51abf6040f0228b9"} Sep 10 14:00:55.299772 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.299712 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podStartSLOduration=1.299694455 podStartE2EDuration="1.299694455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:55.29858107 +0000 UTC m=+1933.367581036" watchObservedRunningTime="2026-09-10 14:00:55.299694455 +0000 UTC m=+1933.368694425" Sep 10 14:00:55.827270 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:55.827235 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:56.833577 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:56.833545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:57.286905 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:57.286874 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:57.287963 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:57.287940 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:00:59.223165 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.223127 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns"] Sep 10 14:00:59.223591 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.223577 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerName="cleanup" Sep 10 14:00:59.223681 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.223593 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" containerName="cleanup" Sep 10 14:00:59.227037 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.227017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.229835 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.229811 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-eb3d6d\"" Sep 10 14:00:59.237452 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.237429 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns"] Sep 10 14:00:59.243073 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.243046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.243206 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.243095 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27fp\" (UniqueName: \"kubernetes.io/projected/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-kube-api-access-w27fp\") pod \"payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.344929 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.344882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.345142 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.344956 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27fp\" (UniqueName: \"kubernetes.io/projected/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-kube-api-access-w27fp\") pod \"payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.347165 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.347134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.361296 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.361264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27fp\" (UniqueName: \"kubernetes.io/projected/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-kube-api-access-w27fp\") pod \"payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.539662 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.539550 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:00:59.817820 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:00:59.817795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns"] Sep 10 14:00:59.819354 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:00:59.819317 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de08ca8_f84d_4454_b98a_47d9fb4d8c64.slice/crio-fdc12906cdd4187db19e9025e979b9e96c430437c77f989cc9f140ea661ff843 WatchSource:0}: Error finding container fdc12906cdd4187db19e9025e979b9e96c430437c77f989cc9f140ea661ff843: Status 404 returned error can't find the container with id fdc12906cdd4187db19e9025e979b9e96c430437c77f989cc9f140ea661ff843 Sep 10 14:01:00.306805 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:00.306769 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" event={"ID":"0de08ca8-f84d-4454-b98a-47d9fb4d8c64","Type":"ContainerStarted","Data":"ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8"} Sep 10 14:01:00.307259 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:00.306823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" event={"ID":"0de08ca8-f84d-4454-b98a-47d9fb4d8c64","Type":"ContainerStarted","Data":"fdc12906cdd4187db19e9025e979b9e96c430437c77f989cc9f140ea661ff843"} Sep 10 14:01:00.307259 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:00.306848 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:01:00.327201 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:00.327133 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" podStartSLOduration=1.327113617 podStartE2EDuration="1.327113617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:01:00.326565107 +0000 UTC m=+1938.395565090" watchObservedRunningTime="2026-09-10 14:01:00.327113617 +0000 UTC m=+1938.396113587" Sep 10 14:01:11.316067 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:11.316033 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:01:17.409218 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:17.409185 2568 generic.go:356] "Generic (PLEG): container finished" podID="000f7a48-3803-4456-97ac-f6a9c78069db" containerID="7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8" exitCode=0 Sep 10 14:01:17.409666 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:17.409258 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" event={"ID":"000f7a48-3803-4456-97ac-f6a9c78069db","Type":"ContainerDied","Data":"7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8"} Sep 10 14:01:17.409753 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:17.409737 2568 scope.go:117] "RemoveContainer" containerID="7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8" Sep 10 14:01:17.411078 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:17.411048 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:01:17.411148 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:17.411123 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:01:17.411189 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:17.411157 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:01:18.416534 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:18.416492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" event={"ID":"000f7a48-3803-4456-97ac-f6a9c78069db","Type":"ContainerStarted","Data":"047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194"} Sep 10 14:01:18.827695 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:18.827651 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:01:18.827888 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:18.827864 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 10 14:01:18.827934 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:18.827914 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 10 14:01:19.827312 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:19.827274 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 10 14:01:19.827759 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:19.827335 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 10 14:01:20.827505 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:20.827457 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 10 14:01:20.827948 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:20.827529 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 10 14:01:21.827812 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:21.827767 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 10 14:01:21.828299 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:21.827834 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 10 14:01:22.827397 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:22.827351 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 10 14:01:22.827592 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:22.827422 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 10 14:01:23.046467 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.046408 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842"] Sep 10 14:01:23.050995 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.050962 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:23.053778 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.053755 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:01:23.054027 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.053756 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:01:23.058705 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.058682 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842"] Sep 10 14:01:23.199185 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.199086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842\" (UID: \"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:23.299852 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.299807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842\" (UID: \"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:23.300552 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.300529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842\" (UID: \"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:23.367252 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.367202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:23.673721 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.673692 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842"] Sep 10 14:01:23.675254 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:01:23.675223 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf85fa2_35cf_46df_b71e_4a5cf39aee2d.slice/crio-bf4cb286b9e5737fbade5d1684dc150577a7088ea6e9bfd9f53720cac6997d87 WatchSource:0}: Error finding container bf4cb286b9e5737fbade5d1684dc150577a7088ea6e9bfd9f53720cac6997d87: Status 404 returned error can't find the container with id bf4cb286b9e5737fbade5d1684dc150577a7088ea6e9bfd9f53720cac6997d87 Sep 10 14:01:23.992730 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.992698 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:01:23.993138 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.993103 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:01:23.993982 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:23.993961 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:01:24.472672 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:24.472630 2568 generic.go:356] "Generic (PLEG): container finished" podID="5cf85fa2-35cf-46df-b71e-4a5cf39aee2d" containerID="598f94301667d8ea1c325163a2b1abc0518a3258a4f9333f4b2adfcfd474b025" exitCode=0 Sep 10 14:01:24.473133 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:24.472762 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" event={"ID":"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d","Type":"ContainerDied","Data":"598f94301667d8ea1c325163a2b1abc0518a3258a4f9333f4b2adfcfd474b025"} Sep 10 14:01:24.473133 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:24.472823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" event={"ID":"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d","Type":"ContainerStarted","Data":"bf4cb286b9e5737fbade5d1684dc150577a7088ea6e9bfd9f53720cac6997d87"} Sep 10 14:01:25.618089 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:25.618062 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:25.724680 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:25.724648 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d-maas-api-service-ca\") pod \"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d\" (UID: \"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d\") " Sep 10 14:01:25.725027 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:25.725002 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cf85fa2-35cf-46df-b71e-4a5cf39aee2d" (UID: "5cf85fa2-35cf-46df-b71e-4a5cf39aee2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:25.832686 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:25.832573 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:26.486074 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:26.486041 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" Sep 10 14:01:26.486074 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:26.486050 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842" event={"ID":"5cf85fa2-35cf-46df-b71e-4a5cf39aee2d","Type":"ContainerDied","Data":"bf4cb286b9e5737fbade5d1684dc150577a7088ea6e9bfd9f53720cac6997d87"} Sep 10 14:01:26.486279 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:26.486099 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf4cb286b9e5737fbade5d1684dc150577a7088ea6e9bfd9f53720cac6997d87" Sep 10 14:01:29.532734 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.532694 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns"] Sep 10 14:01:29.533546 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.533487 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" podUID="0de08ca8-f84d-4454-b98a-47d9fb4d8c64" containerName="payload-processing" containerID="cri-o://ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8" gracePeriod=30 Sep 10 14:01:29.797069 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.797011 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:01:29.875618 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.874592 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w27fp\" (UniqueName: \"kubernetes.io/projected/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-kube-api-access-w27fp\") pod \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " Sep 10 14:01:29.875618 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.874750 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-plugins-config-volume\") pod \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\" (UID: \"0de08ca8-f84d-4454-b98a-47d9fb4d8c64\") " Sep 10 14:01:29.875883 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.875695 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0de08ca8-f84d-4454-b98a-47d9fb4d8c64" (UID: "0de08ca8-f84d-4454-b98a-47d9fb4d8c64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:29.879515 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.879478 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-kube-api-access-w27fp" (OuterVolumeSpecName: "kube-api-access-w27fp") pod "0de08ca8-f84d-4454-b98a-47d9fb4d8c64" (UID: "0de08ca8-f84d-4454-b98a-47d9fb4d8c64"). InnerVolumeSpecName "kube-api-access-w27fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:29.955565 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.955528 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842"] Sep 10 14:01:29.973481 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.973450 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb3d6d-fr842"] Sep 10 14:01:29.976536 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.976511 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27fp\" (UniqueName: \"kubernetes.io/projected/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-kube-api-access-w27fp\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:29.976657 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:29.976538 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de08ca8-f84d-4454-b98a-47d9fb4d8c64-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:30.424167 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.424133 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf85fa2-35cf-46df-b71e-4a5cf39aee2d" path="/var/lib/kubelet/pods/5cf85fa2-35cf-46df-b71e-4a5cf39aee2d/volumes" Sep 10 14:01:30.509939 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.509906 2568 generic.go:356] "Generic (PLEG): container finished" podID="0de08ca8-f84d-4454-b98a-47d9fb4d8c64" containerID="ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8" exitCode=0 Sep 10 14:01:30.510109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.509948 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" Sep 10 14:01:30.510109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.509992 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" event={"ID":"0de08ca8-f84d-4454-b98a-47d9fb4d8c64","Type":"ContainerDied","Data":"ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8"} Sep 10 14:01:30.510109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.510048 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns" event={"ID":"0de08ca8-f84d-4454-b98a-47d9fb4d8c64","Type":"ContainerDied","Data":"fdc12906cdd4187db19e9025e979b9e96c430437c77f989cc9f140ea661ff843"} Sep 10 14:01:30.510109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.510071 2568 scope.go:117] "RemoveContainer" containerID="ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8" Sep 10 14:01:30.519075 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.519052 2568 scope.go:117] "RemoveContainer" containerID="ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8" Sep 10 14:01:30.519357 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:01:30.519338 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8\": container with ID starting with ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8 not found: ID does not exist" containerID="ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8" Sep 10 14:01:30.519413 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.519366 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8"} err="failed to get container status \"ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8\": rpc error: code = NotFound desc = could not find container \"ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8\": container with ID starting with ab9c3e619eb9fb05f00ef6bbac913220be17c11bdb62eeed4551289bd7d205d8 not found: ID does not exist" Sep 10 14:01:30.550885 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.550857 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns"] Sep 10 14:01:30.569434 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:30.569405 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb3d6d-54dd57c55-gc4ns"] Sep 10 14:01:32.424557 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:32.424526 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de08ca8-f84d-4454-b98a-47d9fb4d8c64" path="/var/lib/kubelet/pods/0de08ca8-f84d-4454-b98a-47d9fb4d8c64/volumes" Sep 10 14:01:38.313178 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:38.313084 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb"] Sep 10 14:01:38.313627 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:38.313339 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" containerID="cri-o://047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194" gracePeriod=30 Sep 10 14:01:38.995616 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:38.995558 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 14:01:38.995802 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:38.995665 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 14:01:42.054331 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054283 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4"] Sep 10 14:01:42.054776 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054726 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf85fa2-35cf-46df-b71e-4a5cf39aee2d" containerName="revoke-keys" Sep 10 14:01:42.054776 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054739 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf85fa2-35cf-46df-b71e-4a5cf39aee2d" containerName="revoke-keys" Sep 10 14:01:42.054776 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054755 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de08ca8-f84d-4454-b98a-47d9fb4d8c64" containerName="payload-processing" Sep 10 14:01:42.054776 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054761 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de08ca8-f84d-4454-b98a-47d9fb4d8c64" containerName="payload-processing" Sep 10 14:01:42.054935 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054834 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf85fa2-35cf-46df-b71e-4a5cf39aee2d" containerName="revoke-keys" Sep 10 14:01:42.054935 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.054844 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de08ca8-f84d-4454-b98a-47d9fb4d8c64" containerName="payload-processing" Sep 10 14:01:42.059319 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.059301 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:42.062341 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.062318 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:01:42.063073 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.063050 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:01:42.075719 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.075694 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4"] Sep 10 14:01:42.190263 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.190216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e56d3b-c475-46e5-a1fe-31377706b649-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4\" (UID: \"50e56d3b-c475-46e5-a1fe-31377706b649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:42.291005 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.290967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e56d3b-c475-46e5-a1fe-31377706b649-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4\" (UID: \"50e56d3b-c475-46e5-a1fe-31377706b649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:42.293808 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.293786 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:01:42.301713 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.301683 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e56d3b-c475-46e5-a1fe-31377706b649-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4\" (UID: \"50e56d3b-c475-46e5-a1fe-31377706b649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:42.372629 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.372516 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:01:42.379699 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.379675 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:42.660590 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:42.660557 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4"] Sep 10 14:01:42.661463 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:01:42.661434 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e56d3b_c475_46e5_a1fe_31377706b649.slice/crio-802874069c1cc8d7b14732d4d0367c2603543479fdb325c662c3d2a0fea1526c WatchSource:0}: Error finding container 802874069c1cc8d7b14732d4d0367c2603543479fdb325c662c3d2a0fea1526c: Status 404 returned error can't find the container with id 802874069c1cc8d7b14732d4d0367c2603543479fdb325c662c3d2a0fea1526c Sep 10 14:01:43.569214 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.569185 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:01:43.585287 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.585204 2568 generic.go:356] "Generic (PLEG): container finished" podID="50e56d3b-c475-46e5-a1fe-31377706b649" containerID="55f6b14dc07b65832d0b46d547159cb85b73461a92fe8cf4ae60501ff9aa542b" exitCode=0 Sep 10 14:01:43.585450 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.585288 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" event={"ID":"50e56d3b-c475-46e5-a1fe-31377706b649","Type":"ContainerDied","Data":"55f6b14dc07b65832d0b46d547159cb85b73461a92fe8cf4ae60501ff9aa542b"} Sep 10 14:01:43.585450 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.585335 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" event={"ID":"50e56d3b-c475-46e5-a1fe-31377706b649","Type":"ContainerStarted","Data":"802874069c1cc8d7b14732d4d0367c2603543479fdb325c662c3d2a0fea1526c"} Sep 10 14:01:43.587070 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.587046 2568 generic.go:356] "Generic (PLEG): container finished" podID="000f7a48-3803-4456-97ac-f6a9c78069db" containerID="047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194" exitCode=0 Sep 10 14:01:43.587204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.587088 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" Sep 10 14:01:43.587204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.587128 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" event={"ID":"000f7a48-3803-4456-97ac-f6a9c78069db","Type":"ContainerDied","Data":"047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194"} Sep 10 14:01:43.587204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.587180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb" event={"ID":"000f7a48-3803-4456-97ac-f6a9c78069db","Type":"ContainerDied","Data":"0d88ab1d84881223dee377d1508f2decc5e6318c1ec6b43c51abf6040f0228b9"} Sep 10 14:01:43.587322 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.587205 2568 scope.go:117] "RemoveContainer" containerID="047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194" Sep 10 14:01:43.596361 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.596336 2568 scope.go:117] "RemoveContainer" containerID="7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8" Sep 10 14:01:43.607551 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.607527 2568 scope.go:117] "RemoveContainer" containerID="047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194" Sep 10 14:01:43.607892 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:01:43.607867 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194\": container with ID starting with 047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194 not found: ID does not exist" containerID="047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194" Sep 10 14:01:43.607999 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.607899 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194"} err="failed to get container status \"047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194\": rpc error: code = NotFound desc = could not find container \"047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194\": container with ID starting with 047b39671da301d24370f3df86824943e975f5d0f3f8958c31729b26397f2194 not found: ID does not exist" Sep 10 14:01:43.607999 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.607917 2568 scope.go:117] "RemoveContainer" containerID="7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8" Sep 10 14:01:43.608252 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:01:43.608223 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8\": container with ID starting with 7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8 not found: ID does not exist" containerID="7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8" Sep 10 14:01:43.608323 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.608258 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8"} err="failed to get container status \"7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8\": rpc error: code = NotFound desc = could not find container \"7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8\": container with ID starting with 7b397a29ea42783027fa8d7fad22061198fbd07681e0a4a081391f133d17dfe8 not found: ID does not exist" Sep 10 14:01:43.704897 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.704856 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f7a48-3803-4456-97ac-f6a9c78069db-istiod-ca-cert\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.704915 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-envoy\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.704998 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-istio-token\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705036 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-credential-socket\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705073 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f7a48-3803-4456-97ac-f6a9c78069db-istio-podinfo\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705109 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705097 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-884j7\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-kube-api-access-884j7\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705383 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705135 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-data\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705383 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705187 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-socket\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705383 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705224 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-certs\") pod \"000f7a48-3803-4456-97ac-f6a9c78069db\" (UID: \"000f7a48-3803-4456-97ac-f6a9c78069db\") " Sep 10 14:01:43.705383 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705328 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000f7a48-3803-4456-97ac-f6a9c78069db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:43.705650 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705622 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:43.705762 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705715 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:43.705889 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705837 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:43.705956 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705893 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f7a48-3803-4456-97ac-f6a9c78069db-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.705956 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705907 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-data" (OuterVolumeSpecName: "istio-data") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:43.705956 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705914 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.705956 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.705931 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.707404 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.707367 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:43.707404 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.707392 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/000f7a48-3803-4456-97ac-f6a9c78069db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:01:43.707681 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.707665 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-istio-token" (OuterVolumeSpecName: "istio-token") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:43.707755 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.707733 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-kube-api-access-884j7" (OuterVolumeSpecName: "kube-api-access-884j7") pod "000f7a48-3803-4456-97ac-f6a9c78069db" (UID: "000f7a48-3803-4456-97ac-f6a9c78069db"). InnerVolumeSpecName "kube-api-access-884j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:43.806528 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.806491 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.806528 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.806526 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.806807 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.806540 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f7a48-3803-4456-97ac-f6a9c78069db-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.806807 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.806556 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-884j7\" (UniqueName: \"kubernetes.io/projected/000f7a48-3803-4456-97ac-f6a9c78069db-kube-api-access-884j7\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.806807 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.806568 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.806807 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.806579 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f7a48-3803-4456-97ac-f6a9c78069db-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:43.923616 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.923570 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb"] Sep 10 14:01:43.934534 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:43.934502 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eb3d6d-openshift-default-659c55756d-clkxb"] Sep 10 14:01:44.424268 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:44.424224 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" path="/var/lib/kubelet/pods/000f7a48-3803-4456-97ac-f6a9c78069db/volumes" Sep 10 14:01:44.739363 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:44.739334 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:44.814818 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:44.814783 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e56d3b-c475-46e5-a1fe-31377706b649-maas-api-service-ca\") pod \"50e56d3b-c475-46e5-a1fe-31377706b649\" (UID: \"50e56d3b-c475-46e5-a1fe-31377706b649\") " Sep 10 14:01:44.815197 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:44.815162 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e56d3b-c475-46e5-a1fe-31377706b649-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50e56d3b-c475-46e5-a1fe-31377706b649" (UID: "50e56d3b-c475-46e5-a1fe-31377706b649"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:44.915721 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:44.915683 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e56d3b-c475-46e5-a1fe-31377706b649-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:01:45.601056 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:45.601023 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" Sep 10 14:01:45.601056 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:45.601046 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4" event={"ID":"50e56d3b-c475-46e5-a1fe-31377706b649","Type":"ContainerDied","Data":"802874069c1cc8d7b14732d4d0367c2603543479fdb325c662c3d2a0fea1526c"} Sep 10 14:01:45.601350 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:45.601088 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="802874069c1cc8d7b14732d4d0367c2603543479fdb325c662c3d2a0fea1526c" Sep 10 14:01:58.203870 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:58.203832 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4"] Sep 10 14:01:58.219003 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:58.218976 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f01c7-b8kg4"] Sep 10 14:01:58.425158 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:01:58.425122 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e56d3b-c475-46e5-a1fe-31377706b649" path="/var/lib/kubelet/pods/50e56d3b-c475-46e5-a1fe-31377706b649/volumes" Sep 10 14:02:49.922614 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:49.922567 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8"] Sep 10 14:02:49.923136 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:49.922866 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" containerID="cri-o://89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903" gracePeriod=30 Sep 10 14:02:50.276246 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276207 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v"] Sep 10 14:02:50.276634 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276621 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e56d3b-c475-46e5-a1fe-31377706b649" containerName="revoke-keys" Sep 10 14:02:50.276686 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276636 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e56d3b-c475-46e5-a1fe-31377706b649" containerName="revoke-keys" Sep 10 14:02:50.276686 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276659 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" Sep 10 14:02:50.276686 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276665 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" Sep 10 14:02:50.276686 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276672 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" Sep 10 14:02:50.276686 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276678 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" Sep 10 14:02:50.276849 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276742 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" Sep 10 14:02:50.276849 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276752 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="000f7a48-3803-4456-97ac-f6a9c78069db" containerName="istio-proxy" Sep 10 14:02:50.276849 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.276760 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e56d3b-c475-46e5-a1fe-31377706b649" containerName="revoke-keys" Sep 10 14:02:50.285745 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.285705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:50.288447 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.288391 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:02:50.288577 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.288397 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:02:50.289148 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.289110 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v"] Sep 10 14:02:50.430202 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.430167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e0f679-b620-4084-8b9c-c5bf15f9e1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v\" (UID: \"74e0f679-b620-4084-8b9c-c5bf15f9e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:50.531003 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.530894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e0f679-b620-4084-8b9c-c5bf15f9e1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v\" (UID: \"74e0f679-b620-4084-8b9c-c5bf15f9e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:50.531572 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.531554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e0f679-b620-4084-8b9c-c5bf15f9e1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v\" (UID: \"74e0f679-b620-4084-8b9c-c5bf15f9e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:50.597827 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.597799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:50.905095 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:02:50.905060 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e0f679_b620_4084_8b9c_c5bf15f9e1fd.slice/crio-f5e35debceb40813f42963cfe6791065326648b2d1fa1124762724f4ef271274 WatchSource:0}: Error finding container f5e35debceb40813f42963cfe6791065326648b2d1fa1124762724f4ef271274: Status 404 returned error can't find the container with id f5e35debceb40813f42963cfe6791065326648b2d1fa1124762724f4ef271274 Sep 10 14:02:50.909921 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.909895 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v"] Sep 10 14:02:50.974704 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:50.974676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" event={"ID":"74e0f679-b620-4084-8b9c-c5bf15f9e1fd","Type":"ContainerStarted","Data":"f5e35debceb40813f42963cfe6791065326648b2d1fa1124762724f4ef271274"} Sep 10 14:02:51.981405 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:51.981373 2568 generic.go:356] "Generic (PLEG): container finished" podID="74e0f679-b620-4084-8b9c-c5bf15f9e1fd" containerID="99f9d77d0bbf848bcef9858e45132d1571fb47ba7cddaf3d092c11ed0140e15f" exitCode=0 Sep 10 14:02:51.981819 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:51.981459 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" event={"ID":"74e0f679-b620-4084-8b9c-c5bf15f9e1fd","Type":"ContainerDied","Data":"99f9d77d0bbf848bcef9858e45132d1571fb47ba7cddaf3d092c11ed0140e15f"} Sep 10 14:02:53.123549 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.123520 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:53.256969 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.256880 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e0f679-b620-4084-8b9c-c5bf15f9e1fd-maas-api-service-ca\") pod \"74e0f679-b620-4084-8b9c-c5bf15f9e1fd\" (UID: \"74e0f679-b620-4084-8b9c-c5bf15f9e1fd\") " Sep 10 14:02:53.257236 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.257215 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e0f679-b620-4084-8b9c-c5bf15f9e1fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74e0f679-b620-4084-8b9c-c5bf15f9e1fd" (UID: "74e0f679-b620-4084-8b9c-c5bf15f9e1fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:02:53.358568 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.358525 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e0f679-b620-4084-8b9c-c5bf15f9e1fd-maas-api-service-ca\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:53.994527 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.994488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" event={"ID":"74e0f679-b620-4084-8b9c-c5bf15f9e1fd","Type":"ContainerDied","Data":"f5e35debceb40813f42963cfe6791065326648b2d1fa1124762724f4ef271274"} Sep 10 14:02:53.994527 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.994512 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v" Sep 10 14:02:53.994770 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:53.994552 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e35debceb40813f42963cfe6791065326648b2d1fa1124762724f4ef271274" Sep 10 14:02:55.166483 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.166461 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 14:02:55.276823 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.276790 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47d032e-e443-49b2-bacf-755c5ad4511b-istiod-ca-cert\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277031 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.276853 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-data\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277031 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.276893 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-certs\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277031 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.276929 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-socket\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277031 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.276955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swwz5\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-kube-api-access-swwz5\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277031 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.276983 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-token\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277309 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277065 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-credential-socket\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277309 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277114 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-podinfo\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277309 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277151 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-envoy\") pod \"b47d032e-e443-49b2-bacf-755c5ad4511b\" (UID: \"b47d032e-e443-49b2-bacf-755c5ad4511b\") " Sep 10 14:02:55.277309 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277215 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:02:55.277309 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277244 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:02:55.277533 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277336 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47d032e-e443-49b2-bacf-755c5ad4511b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:02:55.277533 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277365 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:02:55.277656 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277628 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:02:55.277701 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277635 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.277701 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277670 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.277701 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277682 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47d032e-e443-49b2-bacf-755c5ad4511b-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.277701 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.277691 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.279767 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.279735 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:02:55.279767 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.279742 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:02:55.279933 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.279822 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-kube-api-access-swwz5" (OuterVolumeSpecName: "kube-api-access-swwz5") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "kube-api-access-swwz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:02:55.279933 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.279830 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b47d032e-e443-49b2-bacf-755c5ad4511b" (UID: "b47d032e-e443-49b2-bacf-755c5ad4511b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:02:55.378782 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.378744 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.378782 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.378780 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwz5\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-kube-api-access-swwz5\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.379070 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.378793 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.379070 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.378805 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:55.379070 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:55.378815 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47d032e-e443-49b2-bacf-755c5ad4511b-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:02:56.009252 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.009213 2568 generic.go:356] "Generic (PLEG): container finished" podID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerID="89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903" exitCode=0 Sep 10 14:02:56.009451 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.009276 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" Sep 10 14:02:56.009451 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.009295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" event={"ID":"b47d032e-e443-49b2-bacf-755c5ad4511b","Type":"ContainerDied","Data":"89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903"} Sep 10 14:02:56.009451 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.009351 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8" event={"ID":"b47d032e-e443-49b2-bacf-755c5ad4511b","Type":"ContainerDied","Data":"c7f80e366e46d03411e1e6ea87dbec4b29fecb1ad8e735f63634b58bbeb49844"} Sep 10 14:02:56.009451 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.009374 2568 scope.go:117] "RemoveContainer" containerID="89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903" Sep 10 14:02:56.019393 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.019374 2568 scope.go:117] "RemoveContainer" containerID="8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596" Sep 10 14:02:56.027778 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.027761 2568 scope.go:117] "RemoveContainer" containerID="89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903" Sep 10 14:02:56.028046 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:02:56.028028 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903\": container with ID starting with 89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903 not found: ID does not exist" containerID="89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903" Sep 10 14:02:56.028085 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.028055 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903"} err="failed to get container status \"89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903\": rpc error: code = NotFound desc = could not find container \"89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903\": container with ID starting with 89e831effa8db35e202f586562d88cb46a11daa9bb9fde4efe96f1798e0d8903 not found: ID does not exist" Sep 10 14:02:56.028085 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.028074 2568 scope.go:117] "RemoveContainer" containerID="8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596" Sep 10 14:02:56.028313 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:02:56.028296 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596\": container with ID starting with 8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596 not found: ID does not exist" containerID="8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596" Sep 10 14:02:56.028352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.028318 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596"} err="failed to get container status \"8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596\": rpc error: code = NotFound desc = could not find container \"8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596\": container with ID starting with 8525d18a1c8410a04088cbf58fb9894b9e264dfeee40a4686e591abeb3b98596 not found: ID does not exist" Sep 10 14:02:56.053493 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.053466 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8"] Sep 10 14:02:56.061534 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.061510 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f01c7-openshift-default-7d6f4b4cc4-85ln8"] Sep 10 14:02:56.424631 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:02:56.424540 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" path="/var/lib/kubelet/pods/b47d032e-e443-49b2-bacf-755c5ad4511b/volumes" Sep 10 14:03:01.264485 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.264444 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7"] Sep 10 14:03:01.268349 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.268319 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0741a-29817480-lnxn7"] Sep 10 14:03:01.313545 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.313508 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm"] Sep 10 14:03:01.313855 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.313800 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" podUID="71cc6a20-0222-42c2-a6f5-a5258135e652" containerName="payload-processing" containerID="cri-o://386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d" gracePeriod=30 Sep 10 14:03:01.568403 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.568376 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 14:03:01.740956 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.740911 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nvgv\" (UniqueName: \"kubernetes.io/projected/71cc6a20-0222-42c2-a6f5-a5258135e652-kube-api-access-4nvgv\") pod \"71cc6a20-0222-42c2-a6f5-a5258135e652\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " Sep 10 14:03:01.741168 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.740981 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cc6a20-0222-42c2-a6f5-a5258135e652-plugins-config-volume\") pod \"71cc6a20-0222-42c2-a6f5-a5258135e652\" (UID: \"71cc6a20-0222-42c2-a6f5-a5258135e652\") " Sep 10 14:03:01.741331 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.741305 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cc6a20-0222-42c2-a6f5-a5258135e652-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71cc6a20-0222-42c2-a6f5-a5258135e652" (UID: "71cc6a20-0222-42c2-a6f5-a5258135e652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:01.743440 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.743411 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cc6a20-0222-42c2-a6f5-a5258135e652-kube-api-access-4nvgv" (OuterVolumeSpecName: "kube-api-access-4nvgv") pod "71cc6a20-0222-42c2-a6f5-a5258135e652" (UID: "71cc6a20-0222-42c2-a6f5-a5258135e652"). InnerVolumeSpecName "kube-api-access-4nvgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:01.842154 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.842061 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvgv\" (UniqueName: \"kubernetes.io/projected/71cc6a20-0222-42c2-a6f5-a5258135e652-kube-api-access-4nvgv\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:01.842154 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.842100 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cc6a20-0222-42c2-a6f5-a5258135e652-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:01.856620 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.856565 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v"] Sep 10 14:03:01.868268 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:01.868239 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0741a-fcp8v"] Sep 10 14:03:02.045928 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.045894 2568 generic.go:356] "Generic (PLEG): container finished" podID="71cc6a20-0222-42c2-a6f5-a5258135e652" containerID="386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d" exitCode=0 Sep 10 14:03:02.045928 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.045934 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" event={"ID":"71cc6a20-0222-42c2-a6f5-a5258135e652","Type":"ContainerDied","Data":"386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d"} Sep 10 14:03:02.046204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.045952 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" Sep 10 14:03:02.046204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.045973 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm" event={"ID":"71cc6a20-0222-42c2-a6f5-a5258135e652","Type":"ContainerDied","Data":"da8e473fbad2f04238e999e48210dd69c42709ef7ea3650aef35996be3f704d9"} Sep 10 14:03:02.046204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.045993 2568 scope.go:117] "RemoveContainer" containerID="386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d" Sep 10 14:03:02.055842 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.055821 2568 scope.go:117] "RemoveContainer" containerID="386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d" Sep 10 14:03:02.056160 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:02.056138 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d\": container with ID starting with 386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d not found: ID does not exist" containerID="386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d" Sep 10 14:03:02.056208 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.056170 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d"} err="failed to get container status \"386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d\": rpc error: code = NotFound desc = could not find container \"386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d\": container with ID starting with 386fdbc4da39029f3439775670310e1c293a08cf0341e0ce7bdb438118a7db1d not found: ID does not exist" Sep 10 14:03:02.083921 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.083892 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm"] Sep 10 14:03:02.097712 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.097638 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0741a-5966484fbf-7pkbm"] Sep 10 14:03:02.424506 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.424426 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71cc6a20-0222-42c2-a6f5-a5258135e652" path="/var/lib/kubelet/pods/71cc6a20-0222-42c2-a6f5-a5258135e652/volumes" Sep 10 14:03:02.424884 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.424801 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e0f679-b620-4084-8b9c-c5bf15f9e1fd" path="/var/lib/kubelet/pods/74e0f679-b620-4084-8b9c-c5bf15f9e1fd/volumes" Sep 10 14:03:02.425052 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:02.425039 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0292118-7b75-4d46-bc1b-429d2fb76b5b" path="/var/lib/kubelet/pods/e0292118-7b75-4d46-bc1b-429d2fb76b5b/volumes" Sep 10 14:03:22.207864 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.207777 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m"] Sep 10 14:03:22.217808 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.217771 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a11e9-29817480-7h24m"] Sep 10 14:03:22.277673 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.277639 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc"] Sep 10 14:03:22.277940 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.277911 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" podUID="26d9c1ca-b634-4ce3-8525-71edf23713a2" containerName="payload-processing" containerID="cri-o://79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4" gracePeriod=30 Sep 10 14:03:22.426149 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.426104 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490a0ef1-265c-40d2-b458-a430ab2f86c9" path="/var/lib/kubelet/pods/490a0ef1-265c-40d2-b458-a430ab2f86c9/volumes" Sep 10 14:03:22.538585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.538559 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 14:03:22.637165 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.637123 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bswcr\" (UniqueName: \"kubernetes.io/projected/26d9c1ca-b634-4ce3-8525-71edf23713a2-kube-api-access-bswcr\") pod \"26d9c1ca-b634-4ce3-8525-71edf23713a2\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " Sep 10 14:03:22.637357 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.637176 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d9c1ca-b634-4ce3-8525-71edf23713a2-plugins-config-volume\") pod \"26d9c1ca-b634-4ce3-8525-71edf23713a2\" (UID: \"26d9c1ca-b634-4ce3-8525-71edf23713a2\") " Sep 10 14:03:22.637579 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.637544 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d9c1ca-b634-4ce3-8525-71edf23713a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26d9c1ca-b634-4ce3-8525-71edf23713a2" (UID: "26d9c1ca-b634-4ce3-8525-71edf23713a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:22.639273 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.639245 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d9c1ca-b634-4ce3-8525-71edf23713a2-kube-api-access-bswcr" (OuterVolumeSpecName: "kube-api-access-bswcr") pod "26d9c1ca-b634-4ce3-8525-71edf23713a2" (UID: "26d9c1ca-b634-4ce3-8525-71edf23713a2"). InnerVolumeSpecName "kube-api-access-bswcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:22.738335 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.738295 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswcr\" (UniqueName: \"kubernetes.io/projected/26d9c1ca-b634-4ce3-8525-71edf23713a2-kube-api-access-bswcr\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:22.738335 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:22.738329 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d9c1ca-b634-4ce3-8525-71edf23713a2-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:23.169521 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.169490 2568 generic.go:356] "Generic (PLEG): container finished" podID="26d9c1ca-b634-4ce3-8525-71edf23713a2" containerID="79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4" exitCode=0 Sep 10 14:03:23.169734 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.169534 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" Sep 10 14:03:23.169734 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.169580 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" event={"ID":"26d9c1ca-b634-4ce3-8525-71edf23713a2","Type":"ContainerDied","Data":"79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4"} Sep 10 14:03:23.169734 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.169658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc" event={"ID":"26d9c1ca-b634-4ce3-8525-71edf23713a2","Type":"ContainerDied","Data":"8f0bbd93fb869f687182eeea30b44318b516c6663e74a8bf7b8a9aacc6ba8492"} Sep 10 14:03:23.169734 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.169686 2568 scope.go:117] "RemoveContainer" containerID="79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4" Sep 10 14:03:23.179036 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.179016 2568 scope.go:117] "RemoveContainer" containerID="79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4" Sep 10 14:03:23.179380 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:23.179360 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4\": container with ID starting with 79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4 not found: ID does not exist" containerID="79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4" Sep 10 14:03:23.179430 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.179392 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4"} err="failed to get container status \"79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4\": rpc error: code = NotFound desc = could not find container \"79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4\": container with ID starting with 79e2b4fa157a8d948c82ef0107d667ecb6ed464ec085bb6d34c11594b85e07a4 not found: ID does not exist" Sep 10 14:03:23.195421 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.195384 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc"] Sep 10 14:03:23.203820 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:23.203788 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a11e9-75b9f9685f-2nfbc"] Sep 10 14:03:24.424750 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:24.424714 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d9c1ca-b634-4ce3-8525-71edf23713a2" path="/var/lib/kubelet/pods/26d9c1ca-b634-4ce3-8525-71edf23713a2/volumes" Sep 10 14:03:30.270771 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:30.270734 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w"] Sep 10 14:03:30.271275 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:30.270973 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" containerID="cri-o://bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312" gracePeriod=30 Sep 10 14:03:35.538345 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.538278 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 14:03:35.667158 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667116 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-envoy\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667167 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-token\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667206 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istiod-ca-cert\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667241 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-credential-socket\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667288 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-socket\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667313 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-podinfo\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667352 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667353 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-certs\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667370 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrltb\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-kube-api-access-vrltb\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667400 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-data\") pod \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\" (UID: \"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2\") " Sep 10 14:03:35.667728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667548 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:35.667728 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667700 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:35.667941 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667870 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.667941 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667860 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-data" (OuterVolumeSpecName: "istio-data") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:35.667941 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667893 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.668070 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667967 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:35.668070 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.667985 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:35.669590 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.669561 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:03:35.669741 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.669690 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-kube-api-access-vrltb" (OuterVolumeSpecName: "kube-api-access-vrltb") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "kube-api-access-vrltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:35.669863 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.669844 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-token" (OuterVolumeSpecName: "istio-token") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:35.669927 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.669844 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" (UID: "04e3f7e5-d80f-49d5-8f2a-4b7125268ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:35.768859 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768820 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.768859 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768853 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.768859 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768863 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.769116 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768874 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrltb\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-kube-api-access-vrltb\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.769116 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768886 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.769116 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768894 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:35.769116 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:35.768901 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:36.248699 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.248661 2568 generic.go:356] "Generic (PLEG): container finished" podID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerID="bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312" exitCode=0 Sep 10 14:03:36.248882 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.248713 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" Sep 10 14:03:36.248882 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.248741 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" event={"ID":"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2","Type":"ContainerDied","Data":"bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312"} Sep 10 14:03:36.248882 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.248800 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w" event={"ID":"04e3f7e5-d80f-49d5-8f2a-4b7125268ea2","Type":"ContainerDied","Data":"370e17b4d9211f43699e1e97c7b3fff33c816cc85b3bbe30e0a5352aa7717017"} Sep 10 14:03:36.248882 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.248822 2568 scope.go:117] "RemoveContainer" containerID="bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312" Sep 10 14:03:36.258861 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.258836 2568 scope.go:117] "RemoveContainer" containerID="9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf" Sep 10 14:03:36.267897 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.267874 2568 scope.go:117] "RemoveContainer" containerID="bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312" Sep 10 14:03:36.268250 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:36.268228 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312\": container with ID starting with bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312 not found: ID does not exist" containerID="bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312" Sep 10 14:03:36.268300 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.268260 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312"} err="failed to get container status \"bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312\": rpc error: code = NotFound desc = could not find container \"bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312\": container with ID starting with bdec305e16fcb23661b80cef2676d70bcf4f3f5fd7b4850922ab59449f622312 not found: ID does not exist" Sep 10 14:03:36.268300 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.268279 2568 scope.go:117] "RemoveContainer" containerID="9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf" Sep 10 14:03:36.268558 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:36.268540 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf\": container with ID starting with 9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf not found: ID does not exist" containerID="9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf" Sep 10 14:03:36.268626 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.268568 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf"} err="failed to get container status \"9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf\": rpc error: code = NotFound desc = could not find container \"9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf\": container with ID starting with 9b569da6f2d98613376908bcf01315e90a2c75f06cfb5c5fb8131802a6804edf not found: ID does not exist" Sep 10 14:03:36.275036 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.275001 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w"] Sep 10 14:03:36.277426 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.277403 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a11e9-openshift-default-746748bbb8-n6t9w"] Sep 10 14:03:36.424536 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:36.424491 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" path="/var/lib/kubelet/pods/04e3f7e5-d80f-49d5-8f2a-4b7125268ea2/volumes" Sep 10 14:03:41.903778 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:41.903733 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx"] Sep 10 14:03:41.905591 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:41.905570 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a00003-29817480-xnbcx"] Sep 10 14:03:41.960176 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:41.960137 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt"] Sep 10 14:03:41.960439 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:41.960390 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" podUID="64f157ad-37d8-4367-a49a-fdfa6cf5c583" containerName="payload-processing" containerID="cri-o://652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf" gracePeriod=30 Sep 10 14:03:42.218567 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.218535 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 14:03:42.228440 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.228413 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8hs\" (UniqueName: \"kubernetes.io/projected/64f157ad-37d8-4367-a49a-fdfa6cf5c583-kube-api-access-lz8hs\") pod \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " Sep 10 14:03:42.230806 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.230775 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f157ad-37d8-4367-a49a-fdfa6cf5c583-kube-api-access-lz8hs" (OuterVolumeSpecName: "kube-api-access-lz8hs") pod "64f157ad-37d8-4367-a49a-fdfa6cf5c583" (UID: "64f157ad-37d8-4367-a49a-fdfa6cf5c583"). InnerVolumeSpecName "kube-api-access-lz8hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:42.283966 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.283932 2568 generic.go:356] "Generic (PLEG): container finished" podID="64f157ad-37d8-4367-a49a-fdfa6cf5c583" containerID="652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf" exitCode=0 Sep 10 14:03:42.284159 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.283978 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" Sep 10 14:03:42.284159 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.284021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" event={"ID":"64f157ad-37d8-4367-a49a-fdfa6cf5c583","Type":"ContainerDied","Data":"652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf"} Sep 10 14:03:42.284159 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.284080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt" event={"ID":"64f157ad-37d8-4367-a49a-fdfa6cf5c583","Type":"ContainerDied","Data":"4c36969fc520cecf249c9a5ee38ab92a9a1533341edfbaca397c0e9640de7da0"} Sep 10 14:03:42.284159 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.284106 2568 scope.go:117] "RemoveContainer" containerID="652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf" Sep 10 14:03:42.293202 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.293184 2568 scope.go:117] "RemoveContainer" containerID="652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf" Sep 10 14:03:42.293502 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:42.293481 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf\": container with ID starting with 652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf not found: ID does not exist" containerID="652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf" Sep 10 14:03:42.293578 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.293518 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf"} err="failed to get container status \"652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf\": rpc error: code = NotFound desc = could not find container \"652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf\": container with ID starting with 652b8a211c903308efa49843cb90390fa1df9d9f3e4e81ff81786a950a9fddcf not found: ID does not exist" Sep 10 14:03:42.329067 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.329033 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f157ad-37d8-4367-a49a-fdfa6cf5c583-plugins-config-volume\") pod \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\" (UID: \"64f157ad-37d8-4367-a49a-fdfa6cf5c583\") " Sep 10 14:03:42.329288 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.329274 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8hs\" (UniqueName: \"kubernetes.io/projected/64f157ad-37d8-4367-a49a-fdfa6cf5c583-kube-api-access-lz8hs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:42.329393 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.329369 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f157ad-37d8-4367-a49a-fdfa6cf5c583-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64f157ad-37d8-4367-a49a-fdfa6cf5c583" (UID: "64f157ad-37d8-4367-a49a-fdfa6cf5c583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:42.425258 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.425223 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba1fcd5d-dc70-42c6-a56a-b64af934e26b" path="/var/lib/kubelet/pods/ba1fcd5d-dc70-42c6-a56a-b64af934e26b/volumes" Sep 10 14:03:42.429952 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.429929 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f157ad-37d8-4367-a49a-fdfa6cf5c583-plugins-config-volume\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:42.530962 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.530934 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 14:03:42.538455 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.538431 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 14:03:42.539204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.539176 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 14:03:42.546471 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.546447 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 14:03:42.608313 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.608278 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt"] Sep 10 14:03:42.617867 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.617832 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a00003-c6dc88649-28qrt"] Sep 10 14:03:42.760422 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:42.760383 2568 scope.go:117] "RemoveContainer" containerID="c2d3d541c9d829068867ca578ef462cdea3718db28b086772a56dcc5f040c66f" Sep 10 14:03:44.423711 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:44.423677 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f157ad-37d8-4367-a49a-fdfa6cf5c583" path="/var/lib/kubelet/pods/64f157ad-37d8-4367-a49a-fdfa6cf5c583/volumes" Sep 10 14:03:50.375221 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:50.375177 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv"] Sep 10 14:03:50.375731 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:50.375407 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" containerID="cri-o://492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c" gracePeriod=30 Sep 10 14:03:55.626779 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.626750 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 14:03:55.653339 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653230 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-credential-socket\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653339 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653303 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-envoy\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653379 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-data\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653402 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-istio-token\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653441 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e841089e-abd4-444b-bf01-76e757472a53-istio-podinfo\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653468 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-certs\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653526 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e841089e-abd4-444b-bf01-76e757472a53-istiod-ca-cert\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653585 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:55.653944 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653658 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md9sp\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-kube-api-access-md9sp\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.653944 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653684 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-data" (OuterVolumeSpecName: "istio-data") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:55.653944 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653715 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-socket\") pod \"e841089e-abd4-444b-bf01-76e757472a53\" (UID: \"e841089e-abd4-444b-bf01-76e757472a53\") " Sep 10 14:03:55.654114 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653940 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:55.654114 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.653957 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:55.654114 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.654071 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.654114 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.654091 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-credential-socket\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.654114 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.654106 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-data\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.654318 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.654118 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-workload-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.654318 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.654127 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e841089e-abd4-444b-bf01-76e757472a53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:55.655720 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.655692 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e841089e-abd4-444b-bf01-76e757472a53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:03:55.655837 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.655749 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:55.656000 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.655980 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-istio-token" (OuterVolumeSpecName: "istio-token") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:55.656084 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.656048 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-kube-api-access-md9sp" (OuterVolumeSpecName: "kube-api-access-md9sp") pod "e841089e-abd4-444b-bf01-76e757472a53" (UID: "e841089e-abd4-444b-bf01-76e757472a53"). InnerVolumeSpecName "kube-api-access-md9sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:55.754661 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.754624 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-istio-token\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.754661 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.754655 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e841089e-abd4-444b-bf01-76e757472a53-istio-podinfo\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.754661 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.754663 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e841089e-abd4-444b-bf01-76e757472a53-istiod-ca-cert\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.754909 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.754672 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md9sp\" (UniqueName: \"kubernetes.io/projected/e841089e-abd4-444b-bf01-76e757472a53-kube-api-access-md9sp\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:55.754909 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:55.754683 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e841089e-abd4-444b-bf01-76e757472a53-istio-envoy\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:03:56.370004 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.369972 2568 generic.go:356] "Generic (PLEG): container finished" podID="e841089e-abd4-444b-bf01-76e757472a53" containerID="492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c" exitCode=0 Sep 10 14:03:56.370198 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.370027 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" Sep 10 14:03:56.370198 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.370058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" event={"ID":"e841089e-abd4-444b-bf01-76e757472a53","Type":"ContainerDied","Data":"492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c"} Sep 10 14:03:56.370198 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.370118 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv" event={"ID":"e841089e-abd4-444b-bf01-76e757472a53","Type":"ContainerDied","Data":"4e8077074935314951e3a18fdac60c88c2443f49e3a09b304bd6c0c9f46d21f5"} Sep 10 14:03:56.370198 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.370141 2568 scope.go:117] "RemoveContainer" containerID="492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c" Sep 10 14:03:56.380011 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.379991 2568 scope.go:117] "RemoveContainer" containerID="bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0" Sep 10 14:03:56.388804 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.388784 2568 scope.go:117] "RemoveContainer" containerID="492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c" Sep 10 14:03:56.389122 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:56.389102 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c\": container with ID starting with 492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c not found: ID does not exist" containerID="492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c" Sep 10 14:03:56.389172 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.389131 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c"} err="failed to get container status \"492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c\": rpc error: code = NotFound desc = could not find container \"492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c\": container with ID starting with 492c592aaa4c1f65d095f4dc04ff621269326a343802ef4b27acb2ef0d161a9c not found: ID does not exist" Sep 10 14:03:56.389172 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.389151 2568 scope.go:117] "RemoveContainer" containerID="bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0" Sep 10 14:03:56.389398 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:03:56.389379 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0\": container with ID starting with bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0 not found: ID does not exist" containerID="bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0" Sep 10 14:03:56.389444 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.389404 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0"} err="failed to get container status \"bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0\": rpc error: code = NotFound desc = could not find container \"bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0\": container with ID starting with bc1b4af1a335c804b301b1769adb1a0dd5596d8cf14c22d2d597fc100097e7c0 not found: ID does not exist" Sep 10 14:03:56.405571 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.405533 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv"] Sep 10 14:03:56.417341 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.417308 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a00003-openshift-default-6dd4c47886-9vpgv"] Sep 10 14:03:56.423611 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:03:56.423571 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e841089e-abd4-444b-bf01-76e757472a53" path="/var/lib/kubelet/pods/e841089e-abd4-444b-bf01-76e757472a53/volumes" Sep 10 14:04:42.807491 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:04:42.807399 2568 scope.go:117] "RemoveContainer" containerID="b62db0c1b998f57f07f1522ade08981fbab7b9a3c86a2f0be90694672331cdcc" Sep 10 14:06:42.840363 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:06:42.840328 2568 scope.go:117] "RemoveContainer" containerID="cc0b54b03f0abfab206febc3f82817e96d0e33fe78095f3de48a4efa430d4474" Sep 10 14:06:42.848893 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:06:42.848869 2568 scope.go:117] "RemoveContainer" containerID="66241bb6091c7292d04576fd320705645b6b3ebcddc7ea5d5f8fad39fd446d24" Sep 10 14:06:42.856562 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:06:42.856539 2568 scope.go:117] "RemoveContainer" containerID="11966c6e1f289e09bf3180e97a6aaca5cc9a208013fde54edb67e665761312b9" Sep 10 14:07:42.881658 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:07:42.881561 2568 scope.go:117] "RemoveContainer" containerID="55f6b14dc07b65832d0b46d547159cb85b73461a92fe8cf4ae60501ff9aa542b" Sep 10 14:07:42.889900 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:07:42.889881 2568 scope.go:117] "RemoveContainer" containerID="598f94301667d8ea1c325163a2b1abc0518a3258a4f9333f4b2adfcfd474b025" Sep 10 14:08:42.562587 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:42.562560 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 14:08:42.569885 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:42.569864 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 14:08:42.571121 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:42.571099 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 14:08:42.577651 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:42.577633 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnwpz_1ac55583-190e-4a8f-8256-57e495e3ee98/kube-multus/1.log" Sep 10 14:08:45.735192 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735162 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj"] Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735483 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735494 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735507 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735512 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735518 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735524 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735533 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e0f679-b620-4084-8b9c-c5bf15f9e1fd" containerName="revoke-keys" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735538 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e0f679-b620-4084-8b9c-c5bf15f9e1fd" containerName="revoke-keys" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735545 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" Sep 10 14:08:45.735548 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735550 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735558 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735563 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735571 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f157ad-37d8-4367-a49a-fdfa6cf5c583" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735577 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f157ad-37d8-4367-a49a-fdfa6cf5c583" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735587 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cc6a20-0222-42c2-a6f5-a5258135e652" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735592 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cc6a20-0222-42c2-a6f5-a5258135e652" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735613 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d9c1ca-b634-4ce3-8525-71edf23713a2" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735618 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d9c1ca-b634-4ce3-8525-71edf23713a2" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735675 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735683 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d9c1ca-b634-4ce3-8525-71edf23713a2" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735690 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735696 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e3f7e5-d80f-49d5-8f2a-4b7125268ea2" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735701 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f157ad-37d8-4367-a49a-fdfa6cf5c583" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735707 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e0f679-b620-4084-8b9c-c5bf15f9e1fd" containerName="revoke-keys" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735713 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71cc6a20-0222-42c2-a6f5-a5258135e652" containerName="payload-processing" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735719 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" Sep 10 14:08:45.735879 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.735724 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" Sep 10 14:08:45.738675 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.738657 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:45.742673 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.742650 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-878b46e6-kserve-self-signed-certs\"" Sep 10 14:08:45.755462 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.755439 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj"] Sep 10 14:08:45.902385 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.902355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-model-cache\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:45.902552 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.902394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-home\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:45.902552 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.902483 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-dshm\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:45.902552 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.902506 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48qd\" (UniqueName: \"kubernetes.io/projected/71358cc3-cb99-4c0e-84fa-8b817c21a660-kube-api-access-w48qd\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:45.902552 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.902549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-tmp-dir\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:45.902721 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:45.902585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71358cc3-cb99-4c0e-84fa-8b817c21a660-tls-certs\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.003774 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.003691 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71358cc3-cb99-4c0e-84fa-8b817c21a660-tls-certs\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.003774 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.003729 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-model-cache\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.003774 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.003758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-home\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.004043 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.003803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-dshm\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.004043 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.003821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48qd\" (UniqueName: \"kubernetes.io/projected/71358cc3-cb99-4c0e-84fa-8b817c21a660-kube-api-access-w48qd\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.004043 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.003877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-tmp-dir\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.004204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.004142 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-model-cache\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.004204 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.004180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-tmp-dir\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.004319 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.004279 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-home\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.006031 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.006003 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-dshm\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.006426 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.006405 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71358cc3-cb99-4c0e-84fa-8b817c21a660-tls-certs\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.013000 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.012980 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48qd\" (UniqueName: \"kubernetes.io/projected/71358cc3-cb99-4c0e-84fa-8b817c21a660-kube-api-access-w48qd\") pod \"e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.046933 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.046904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:46.323323 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.323297 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj"] Sep 10 14:08:46.325415 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:08:46.325390 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71358cc3_cb99_4c0e_84fa_8b817c21a660.slice/crio-e1ec7cd5c049e466574c5d81f92225393d9315e6cd04db9935fca68474634300 WatchSource:0}: Error finding container e1ec7cd5c049e466574c5d81f92225393d9315e6cd04db9935fca68474634300: Status 404 returned error can't find the container with id e1ec7cd5c049e466574c5d81f92225393d9315e6cd04db9935fca68474634300 Sep 10 14:08:46.856755 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.856719 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" event={"ID":"71358cc3-cb99-4c0e-84fa-8b817c21a660","Type":"ContainerStarted","Data":"83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d"} Sep 10 14:08:46.857152 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.856764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" event={"ID":"71358cc3-cb99-4c0e-84fa-8b817c21a660","Type":"ContainerStarted","Data":"e1ec7cd5c049e466574c5d81f92225393d9315e6cd04db9935fca68474634300"} Sep 10 14:08:46.878568 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:46.878523 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" podStartSLOduration=1.8785101229999999 podStartE2EDuration="1.878510123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:08:46.876694469 +0000 UTC m=+2404.945694435" watchObservedRunningTime="2026-09-10 14:08:46.878510123 +0000 UTC m=+2404.947510089" Sep 10 14:08:56.047259 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:56.047216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:56.047721 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:56.047478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:56.059806 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:56.059785 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:08:56.916418 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:08:56.916384 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:09:42.918053 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:09:42.917976 2568 scope.go:117] "RemoveContainer" containerID="99f9d77d0bbf848bcef9858e45132d1571fb47ba7cddaf3d092c11ed0140e15f" Sep 10 14:09:48.542885 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:09:48.542848 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj"] Sep 10 14:10:03.562473 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.562426 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" podUID="71358cc3-cb99-4c0e-84fa-8b817c21a660" containerName="main" containerID="cri-o://83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d" gracePeriod=45 Sep 10 14:10:03.807351 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.807326 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:10:03.886659 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886548 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w48qd\" (UniqueName: \"kubernetes.io/projected/71358cc3-cb99-4c0e-84fa-8b817c21a660-kube-api-access-w48qd\") pod \"71358cc3-cb99-4c0e-84fa-8b817c21a660\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " Sep 10 14:10:03.886659 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886626 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-dshm\") pod \"71358cc3-cb99-4c0e-84fa-8b817c21a660\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " Sep 10 14:10:03.886876 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886670 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-model-cache\") pod \"71358cc3-cb99-4c0e-84fa-8b817c21a660\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " Sep 10 14:10:03.886876 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886694 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-tmp-dir\") pod \"71358cc3-cb99-4c0e-84fa-8b817c21a660\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " Sep 10 14:10:03.886876 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886770 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71358cc3-cb99-4c0e-84fa-8b817c21a660-tls-certs\") pod \"71358cc3-cb99-4c0e-84fa-8b817c21a660\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " Sep 10 14:10:03.886876 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886821 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-home\") pod \"71358cc3-cb99-4c0e-84fa-8b817c21a660\" (UID: \"71358cc3-cb99-4c0e-84fa-8b817c21a660\") " Sep 10 14:10:03.887078 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.886943 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-model-cache" (OuterVolumeSpecName: "model-cache") pod "71358cc3-cb99-4c0e-84fa-8b817c21a660" (UID: "71358cc3-cb99-4c0e-84fa-8b817c21a660"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:10:03.887134 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.887048 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71358cc3-cb99-4c0e-84fa-8b817c21a660" (UID: "71358cc3-cb99-4c0e-84fa-8b817c21a660"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:10:03.887134 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.887128 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-model-cache\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:10:03.887245 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.887130 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-home" (OuterVolumeSpecName: "home") pod "71358cc3-cb99-4c0e-84fa-8b817c21a660" (UID: "71358cc3-cb99-4c0e-84fa-8b817c21a660"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:10:03.888964 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.888938 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71358cc3-cb99-4c0e-84fa-8b817c21a660-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71358cc3-cb99-4c0e-84fa-8b817c21a660" (UID: "71358cc3-cb99-4c0e-84fa-8b817c21a660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:10:03.889186 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.889163 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71358cc3-cb99-4c0e-84fa-8b817c21a660-kube-api-access-w48qd" (OuterVolumeSpecName: "kube-api-access-w48qd") pod "71358cc3-cb99-4c0e-84fa-8b817c21a660" (UID: "71358cc3-cb99-4c0e-84fa-8b817c21a660"). InnerVolumeSpecName "kube-api-access-w48qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:10:03.889186 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.889167 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-dshm" (OuterVolumeSpecName: "dshm") pod "71358cc3-cb99-4c0e-84fa-8b817c21a660" (UID: "71358cc3-cb99-4c0e-84fa-8b817c21a660"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:10:03.987486 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.987463 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71358cc3-cb99-4c0e-84fa-8b817c21a660-tls-certs\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:10:03.987486 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.987482 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-home\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:10:03.987486 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.987490 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48qd\" (UniqueName: \"kubernetes.io/projected/71358cc3-cb99-4c0e-84fa-8b817c21a660-kube-api-access-w48qd\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:10:03.987699 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.987498 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-dshm\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:10:03.987699 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:03.987506 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71358cc3-cb99-4c0e-84fa-8b817c21a660-tmp-dir\") on node \"ip-10-0-139-172.ec2.internal\" DevicePath \"\"" Sep 10 14:10:04.232726 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.232693 2568 generic.go:356] "Generic (PLEG): container finished" podID="71358cc3-cb99-4c0e-84fa-8b817c21a660" containerID="83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d" exitCode=0 Sep 10 14:10:04.232891 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.232741 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" Sep 10 14:10:04.232891 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.232777 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" event={"ID":"71358cc3-cb99-4c0e-84fa-8b817c21a660","Type":"ContainerDied","Data":"83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d"} Sep 10 14:10:04.232891 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.232823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj" event={"ID":"71358cc3-cb99-4c0e-84fa-8b817c21a660","Type":"ContainerDied","Data":"e1ec7cd5c049e466574c5d81f92225393d9315e6cd04db9935fca68474634300"} Sep 10 14:10:04.232891 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.232840 2568 scope.go:117] "RemoveContainer" containerID="83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d" Sep 10 14:10:04.241586 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.241570 2568 scope.go:117] "RemoveContainer" containerID="83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d" Sep 10 14:10:04.241885 ip-10-0-139-172 kubenswrapper[2568]: E0910 14:10:04.241866 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d\": container with ID starting with 83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d not found: ID does not exist" containerID="83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d" Sep 10 14:10:04.241949 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.241893 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d"} err="failed to get container status \"83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d\": rpc error: code = NotFound desc = could not find container \"83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d\": container with ID starting with 83192146132fbae826b7fb0c665197395be2cfea2fa99be2089ae305fa87f33d not found: ID does not exist" Sep 10 14:10:04.263180 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.263158 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj"] Sep 10 14:10:04.280050 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.280028 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-878b46e6-kserve-6f9b8f54c-2cgjj"] Sep 10 14:10:04.423852 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:04.423818 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71358cc3-cb99-4c0e-84fa-8b817c21a660" path="/var/lib/kubelet/pods/71358cc3-cb99-4c0e-84fa-8b817c21a660/volumes" Sep 10 14:10:48.469179 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:48.469148 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-z88b8_cb29e624-fbf2-4f5d-a69d-5c1a1c16d648/server/0.log" Sep 10 14:10:48.596346 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:48.596295 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gdssn_d93709e4-ccec-4ec7-9f0c-777b6c32e703/manager/0.log" Sep 10 14:10:48.866388 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:48.866313 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-94r56_6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93/copy-manifests/0.log" Sep 10 14:10:48.874104 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:48.874079 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-94r56_6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93/manager/0.log" Sep 10 14:10:49.589359 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:49.589334 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817465-pvlfg_8232f2c2-ee24-4972-81a8-7fa175c21f74/cleanup/0.log" Sep 10 14:10:51.697891 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:51.697858 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4_7ca44212-723b-43e1-a744-e64bbd11ed44/istio-proxy/0.log" Sep 10 14:10:51.816640 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:51.816616 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9dzjc_b7934da2-9826-4123-b961-bbd86a0b1e4a/discovery/0.log" Sep 10 14:10:52.399864 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:52.399832 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mbq7f_e1be089e-0c86-496e-82b2-592718142acb/payload-processing/0.log" Sep 10 14:10:52.514375 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:52.514339 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64447d557d-vpzh2_9d949756-5384-47bf-a2e0-4380b00b5e5a/router/0.log" Sep 10 14:10:52.969662 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:52.969628 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-sv6cf_579765f9-0be9-448b-90cc-3b4ccdb86ecd/main/0.log" Sep 10 14:10:53.085977 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:10:53.085945 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-v2cz4_6e40bd3c-06f9-41f0-8385-52927b807735/main/0.log" Sep 10 14:11:04.285094 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:04.285043 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ldfgt_21675ea4-e8e5-4530-8b0a-397ae6ba0f92/global-pull-secret-syncer/0.log" Sep 10 14:11:04.453378 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:04.453345 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h68xm_b06c3f96-95c4-47d2-ba93-38a59005cacf/konnectivity-agent/0.log" Sep 10 14:11:04.557411 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:04.557327 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-172.ec2.internal_578987157c44837be8ce53d3e7f8bf0f/haproxy/0.log" Sep 10 14:11:11.078756 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:11.078726 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-q9rnx_91c4e41b-40d7-4042-8d81-b62e0d488e98/cluster-monitoring-operator/0.log" Sep 10 14:11:11.813786 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:11.813757 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trth9_50dbf940-291f-49cf-aacd-90a1b04b3a6a/node-exporter/0.log" Sep 10 14:11:11.878161 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:11.878128 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trth9_50dbf940-291f-49cf-aacd-90a1b04b3a6a/kube-rbac-proxy/0.log" Sep 10 14:11:11.921826 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:11.921802 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trth9_50dbf940-291f-49cf-aacd-90a1b04b3a6a/init-textfile/0.log" Sep 10 14:11:14.173063 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173026 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b"] Sep 10 14:11:14.173576 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173546 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71358cc3-cb99-4c0e-84fa-8b817c21a660" containerName="main" Sep 10 14:11:14.173576 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173563 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71358cc3-cb99-4c0e-84fa-8b817c21a660" containerName="main" Sep 10 14:11:14.173730 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173590 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" Sep 10 14:11:14.173730 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173619 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47d032e-e443-49b2-bacf-755c5ad4511b" containerName="istio-proxy" Sep 10 14:11:14.173836 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173726 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e841089e-abd4-444b-bf01-76e757472a53" containerName="istio-proxy" Sep 10 14:11:14.173836 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.173742 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71358cc3-cb99-4c0e-84fa-8b817c21a660" containerName="main" Sep 10 14:11:14.176935 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.176915 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.180976 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.180956 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vxtc\"/\"kube-root-ca.crt\"" Sep 10 14:11:14.181153 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.181132 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vxtc\"/\"openshift-service-ca.crt\"" Sep 10 14:11:14.181205 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.181149 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5vxtc\"/\"default-dockercfg-mpghg\"" Sep 10 14:11:14.185946 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.185917 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b"] Sep 10 14:11:14.207264 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.207239 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-flw8r_a8dab21e-4671-4586-9934-9f51c1b6dcfe/networking-console-plugin/0.log" Sep 10 14:11:14.209465 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.209444 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-podres\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.209545 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.209489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4kv\" (UniqueName: \"kubernetes.io/projected/e222d682-1eb4-4079-a943-b614e7fa2a9b-kube-api-access-xc4kv\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.209545 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.209519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-proc\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.209640 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.209567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-sys\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.209640 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.209589 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-lib-modules\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310162 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310126 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-podres\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310332 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310172 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4kv\" (UniqueName: \"kubernetes.io/projected/e222d682-1eb4-4079-a943-b614e7fa2a9b-kube-api-access-xc4kv\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310332 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-proc\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310332 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310261 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-sys\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310332 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310307 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-podres\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310332 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310315 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-sys\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310500 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310336 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-proc\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310500 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-lib-modules\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.310500 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.310494 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e222d682-1eb4-4079-a943-b614e7fa2a9b-lib-modules\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.320573 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.320546 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4kv\" (UniqueName: \"kubernetes.io/projected/e222d682-1eb4-4079-a943-b614e7fa2a9b-kube-api-access-xc4kv\") pod \"perf-node-gather-daemonset-mkg2b\" (UID: \"e222d682-1eb4-4079-a943-b614e7fa2a9b\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.487750 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.487719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:14.758669 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.758643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b"] Sep 10 14:11:14.760964 ip-10-0-139-172 kubenswrapper[2568]: W0910 14:11:14.760931 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode222d682_1eb4_4079_a943_b614e7fa2a9b.slice/crio-a2b974b15065629fbcf50f3e9f06fbe65dd1a31f4e20820fea086d2396b42d41 WatchSource:0}: Error finding container a2b974b15065629fbcf50f3e9f06fbe65dd1a31f4e20820fea086d2396b42d41: Status 404 returned error can't find the container with id a2b974b15065629fbcf50f3e9f06fbe65dd1a31f4e20820fea086d2396b42d41 Sep 10 14:11:14.948712 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.948676 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/2.log" Sep 10 14:11:14.956054 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:14.956026 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-56g9k_ab31e74a-1aef-4a39-a7ee-757e5c05e212/console-operator/3.log" Sep 10 14:11:15.575491 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:15.575458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" event={"ID":"e222d682-1eb4-4079-a943-b614e7fa2a9b","Type":"ContainerStarted","Data":"8ceabf399f29684154807ad2cc210039e28543338ff03433d5dfdf112eaa689e"} Sep 10 14:11:15.575907 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:15.575509 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" event={"ID":"e222d682-1eb4-4079-a943-b614e7fa2a9b","Type":"ContainerStarted","Data":"a2b974b15065629fbcf50f3e9f06fbe65dd1a31f4e20820fea086d2396b42d41"} Sep 10 14:11:15.575907 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:15.575585 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:15.594603 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:15.594559 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" podStartSLOduration=1.594544095 podStartE2EDuration="1.594544095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:11:15.592934916 +0000 UTC m=+2553.661934884" watchObservedRunningTime="2026-09-10 14:11:15.594544095 +0000 UTC m=+2553.663544061" Sep 10 14:11:16.236571 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:16.236527 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-67l62_f23db181-9586-4885-ab98-3195ec654d20/volume-data-source-validator/0.log" Sep 10 14:11:17.459936 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:17.459899 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmmz9_246361dc-74c9-4643-8740-456a725d7cb5/dns/0.log" Sep 10 14:11:17.486113 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:17.486087 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmmz9_246361dc-74c9-4643-8740-456a725d7cb5/kube-rbac-proxy/0.log" Sep 10 14:11:17.675753 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:17.675721 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlrlg_7f6499f9-7d26-43d2-a1b9-f86c25c9ce80/dns-node-resolver/0.log" Sep 10 14:11:18.380149 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:18.380115 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cw5l8_c8a458ea-1aef-4cd9-b83e-70477e42d196/node-ca/0.log" Sep 10 14:11:19.542241 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:19.542203 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4_7ca44212-723b-43e1-a744-e64bbd11ed44/istio-proxy/0.log" Sep 10 14:11:19.982560 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:19.982524 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9dzjc_b7934da2-9826-4123-b961-bbd86a0b1e4a/discovery/0.log" Sep 10 14:11:20.178617 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:20.178577 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mbq7f_e1be089e-0c86-496e-82b2-592718142acb/payload-processing/0.log" Sep 10 14:11:20.209080 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:20.209047 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64447d557d-vpzh2_9d949756-5384-47bf-a2e0-4380b00b5e5a/router/0.log" Sep 10 14:11:21.219800 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:21.219763 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhj7q_90f4e27e-5d46-4dfe-bec6-38972d80368b/serve-healthcheck-canary/0.log" Sep 10 14:11:21.589957 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:21.589882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-mkg2b" Sep 10 14:11:21.778283 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:21.778239 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-m5lfk_836f29b4-3ff1-47fc-ab08-28dcdff5144c/insights-operator/1.log" Sep 10 14:11:21.781052 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:21.781028 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-m5lfk_836f29b4-3ff1-47fc-ab08-28dcdff5144c/insights-operator/0.log" Sep 10 14:11:21.983845 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:21.983816 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j5qff_04eae699-fb7a-4431-8f00-7997b5bb047a/kube-rbac-proxy/0.log" Sep 10 14:11:22.010333 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:22.010308 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j5qff_04eae699-fb7a-4431-8f00-7997b5bb047a/exporter/0.log" Sep 10 14:11:22.038923 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:22.038901 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j5qff_04eae699-fb7a-4431-8f00-7997b5bb047a/extractor/0.log" Sep 10 14:11:24.688654 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:24.688618 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4_7ca44212-723b-43e1-a744-e64bbd11ed44/istio-proxy/0.log" Sep 10 14:11:24.950170 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:24.950094 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9dzjc_b7934da2-9826-4123-b961-bbd86a0b1e4a/discovery/0.log" Sep 10 14:11:25.005212 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:25.005184 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-z88b8_cb29e624-fbf2-4f5d-a69d-5c1a1c16d648/server/0.log" Sep 10 14:11:25.150522 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:25.150491 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gdssn_d93709e4-ccec-4ec7-9f0c-777b6c32e703/manager/0.log" Sep 10 14:11:25.213629 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:25.213549 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mbq7f_e1be089e-0c86-496e-82b2-592718142acb/payload-processing/0.log" Sep 10 14:11:25.239661 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:25.239640 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64447d557d-vpzh2_9d949756-5384-47bf-a2e0-4380b00b5e5a/router/0.log" Sep 10 14:11:25.373234 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:25.373201 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-94r56_6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93/manager/0.log" Sep 10 14:11:25.414466 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:25.414440 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-94r56_6e101e8e-0c24-4ee3-83a8-58f0ca2e6f93/copy-manifests/0.log" Sep 10 14:11:26.821298 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:26.821267 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdm5l4_7ca44212-723b-43e1-a744-e64bbd11ed44/istio-proxy/0.log" Sep 10 14:11:26.986009 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:26.985977 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c4cb59d47-knx4g_0b14eae4-9c67-4080-9662-b0370dfddd41/manager/0.log" Sep 10 14:11:27.258491 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:27.258458 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9dzjc_b7934da2-9826-4123-b961-bbd86a0b1e4a/discovery/0.log" Sep 10 14:11:27.416534 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:27.416502 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mbq7f_e1be089e-0c86-496e-82b2-592718142acb/payload-processing/0.log" Sep 10 14:11:27.502391 ip-10-0-139-172 kubenswrapper[2568]: I0910 14:11:27.502358 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64447d557d-vpzh2_9d949756-5384-47bf-a2e0-4380b00b5e5a/router/0.log"