Jul 15 12:24:00.677866 ip-10-0-128-235 systemd[1]: Starting Kubernetes Kubelet... Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.039593 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045616 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045631 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045634 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045637 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045640 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045643 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045653 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:24:01.171498 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045657 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:24:01.107601 ip-10-0-128-235 systemd[1]: Started Kubernetes Kubelet. Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045659 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045663 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045665 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045668 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045671 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045674 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045676 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045680 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045683 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045685 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045688 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045690 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045693 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045697 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045701 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045704 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045707 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045710 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:24:01.172739 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045713 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045716 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045719 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045722 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045725 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045732 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045735 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045738 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045741 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045744 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045746 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045749 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045752 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045755 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045757 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045760 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045762 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045765 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045767 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:24:01.174447 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045770 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045772 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045775 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045778 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045780 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045783 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045787 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045789 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045792 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045795 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045797 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045800 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045804 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045806 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045809 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045812 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045815 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045818 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045820 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045823 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:24:01.203697 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045826 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045828 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045831 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045833 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045836 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045838 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045840 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045843 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045846 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045848 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045851 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045853 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045856 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045858 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045861 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045864 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045866 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045869 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045872 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045875 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:24:01.204280 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045877 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045880 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045885 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045888 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045891 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045893 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045897 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045902 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045905 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045908 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045911 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045914 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045917 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.045919 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048074 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048082 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048085 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048088 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048091 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:24:01.204978 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048094 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048097 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048100 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048103 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048106 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048108 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048111 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048113 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048116 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048119 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048121 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048124 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048130 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048133 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048135 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048138 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048142 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048146 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048148 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:24:01.205522 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048151 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048156 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048160 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048163 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048167 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048170 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048172 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048175 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048178 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048182 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048184 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048187 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048189 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048192 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048194 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048197 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048199 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048202 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048205 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:24:01.206148 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048207 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048211 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048214 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048217 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048220 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048222 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048225 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048229 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048232 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048234 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048237 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048239 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048242 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048245 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048248 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048251 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048253 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048256 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048258 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:24:01.206992 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048261 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048263 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048266 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048269 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048271 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048274 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048276 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048279 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048282 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048284 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048287 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048289 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048291 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048294 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048297 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048299 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048303 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048306 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048308 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:24:01.207767 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048311 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048314 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048317 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048320 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048322 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048325 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048328 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048331 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048333 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048335 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048338 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048340 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048343 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048345 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048348 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048350 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048353 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048355 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048732 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048737 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:24:01.208325 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048740 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048743 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048746 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048748 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048751 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048753 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048756 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048759 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048762 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048766 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048769 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048771 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048774 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048777 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048780 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048782 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048785 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048788 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048791 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048794 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:24:01.209020 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048797 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048799 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048802 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048804 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048806 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048809 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048811 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048814 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048816 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048819 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048821 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048825 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048828 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048831 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048834 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048836 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048838 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048841 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048844 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:24:01.209705 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048846 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048849 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048854 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048857 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048860 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048864 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048866 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048869 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048873 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048876 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048879 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048881 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048885 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048888 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048890 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048893 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048895 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048898 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048900 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048903 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:24:01.210250 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048905 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048908 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048910 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048913 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048915 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048918 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048920 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048922 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048925 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048927 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048930 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048932 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048935 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048937 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048940 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048943 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048946 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048949 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048951 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048954 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:24:01.210837 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048956 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048960 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048962 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048965 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048967 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048970 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048972 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048975 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048977 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048980 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048982 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048985 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048987 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048990 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048992 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048996 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.048999 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049002 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049067 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049073 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049079 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 12:24:01.211396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049084 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049088 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049095 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049099 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049103 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049108 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049111 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049114 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049117 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049120 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049123 2582 flags.go:64] FLAG: --cgroup-root="" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049126 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049129 2582 flags.go:64] FLAG: --client-ca-file="" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049135 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049139 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049142 2582 flags.go:64] FLAG: --cluster-domain="" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049145 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049148 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049151 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049154 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049158 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049161 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049164 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 12:24:01.211963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049167 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049169 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049172 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049176 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049179 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049181 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049185 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049188 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049191 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049193 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049196 2582 flags.go:64] FLAG: --enable-server="true" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049200 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049210 2582 flags.go:64] FLAG: --event-burst="100" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049213 2582 flags.go:64] FLAG: --event-qps="50" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049217 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049220 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049223 2582 flags.go:64] FLAG: --eviction-hard="" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049227 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049230 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049232 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049235 2582 flags.go:64] FLAG: --eviction-soft="" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049238 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049241 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049244 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049247 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 12:24:01.212666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049250 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049253 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049255 2582 flags.go:64] FLAG: --feature-gates="" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049259 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049262 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049265 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049268 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049271 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049274 2582 flags.go:64] FLAG: --help="false" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049277 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049280 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049283 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049286 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049289 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049292 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049295 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049298 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049300 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049304 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049307 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049310 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049314 2582 flags.go:64] FLAG: --kube-reserved="" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049317 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049320 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 12:24:01.213308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049323 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049325 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049328 2582 flags.go:64] FLAG: --lock-file="" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049331 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049334 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049337 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049342 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049345 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049348 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049351 2582 flags.go:64] FLAG: --logging-format="text" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049354 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049357 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049360 2582 flags.go:64] FLAG: --manifest-url="" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049363 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049367 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049370 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049373 2582 flags.go:64] FLAG: --max-pods="110" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049376 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049379 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049382 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049385 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049388 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049391 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049393 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049400 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 12:24:01.214195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049403 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049407 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049410 2582 flags.go:64] FLAG: --pod-cidr="" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049413 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049417 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049422 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049425 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049428 2582 flags.go:64] FLAG: --port="10250" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049431 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049434 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-094bf6223d7744c8b" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049437 2582 flags.go:64] FLAG: --qos-reserved="" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049439 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049442 2582 flags.go:64] FLAG: --register-node="true" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049445 2582 flags.go:64] FLAG: --register-with-taints="" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049448 2582 flags.go:64] FLAG: --registry-burst="10" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049452 2582 flags.go:64] FLAG: --registry-qps="5" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049455 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049458 2582 flags.go:64] FLAG: --reserved-memory="" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049461 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049464 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049467 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049470 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049472 2582 flags.go:64] FLAG: --runonce="false" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049475 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049478 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 12:24:01.215306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049481 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049484 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049486 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049489 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049492 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049495 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049498 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049501 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049505 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049508 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049511 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049514 2582 flags.go:64] FLAG: --system-cgroups="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049517 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049523 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049525 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049528 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049535 2582 flags.go:64] FLAG: --tls-min-version="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049538 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049540 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049543 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049546 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049549 2582 flags.go:64] FLAG: --v="2" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049553 2582 flags.go:64] FLAG: --version="false" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049557 2582 flags.go:64] FLAG: --vmodule="" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049576 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 12:24:01.216434 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.049580 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049707 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049712 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049715 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049718 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049721 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049724 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049726 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049729 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049732 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049734 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049737 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049739 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049742 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049744 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049749 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049751 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049754 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049756 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:24:01.220066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049760 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049764 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049768 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049771 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049773 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049776 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049778 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049781 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049784 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049786 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049788 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049791 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049794 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049796 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049799 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049801 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049804 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049806 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049809 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049813 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:24:01.220918 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049816 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049819 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049822 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049825 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049827 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049830 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049833 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049835 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049839 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049841 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049844 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049846 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049851 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049853 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049856 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049858 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049861 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049864 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049866 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049869 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:24:01.433712 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049871 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049874 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049876 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049879 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049881 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049884 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049887 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049889 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049892 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049894 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049897 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049900 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049903 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049907 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049909 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049912 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049915 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049918 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049920 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:24:01.457224 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049923 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049927 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049930 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049932 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049935 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049939 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049942 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049945 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049947 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049950 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049952 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049955 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049957 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049960 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049963 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049966 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049968 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049971 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049973 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049983 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:24:01.457863 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049985 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.049988 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.050586 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.057543 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.057575 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.057601 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.057609 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057676 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057683 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057686 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057688 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057692 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057695 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057698 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057701 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:24:01.458469 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057704 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057707 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057709 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057712 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057714 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057717 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057720 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057723 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057725 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057728 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057731 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057733 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057736 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057738 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057741 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057744 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057746 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057749 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057751 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057754 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:24:01.458953 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057757 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057766 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057769 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057771 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057774 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057776 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057779 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057781 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057786 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057789 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057792 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057795 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057798 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057800 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057803 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057805 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057808 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057811 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057813 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:24:01.459533 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057816 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057818 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057821 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057823 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057826 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057828 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057831 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057833 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057836 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057840 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057843 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057846 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057851 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057854 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057857 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057866 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057869 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057872 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057874 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057877 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:24:01.460078 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057880 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057882 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057885 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057888 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057890 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057893 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057895 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057898 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057901 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057903 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057906 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057908 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057911 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057913 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057916 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057918 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057921 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057923 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057926 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057929 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:24:01.460678 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057931 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057934 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057936 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057939 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057943 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057945 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057948 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057951 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057960 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057963 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057966 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.057969 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.057975 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058108 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058112 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:24:01.461262 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058115 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058117 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058121 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058124 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058126 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058128 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058131 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058133 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058136 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058139 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058141 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058144 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058146 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058149 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058151 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058154 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058156 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058158 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058161 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058163 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:24:01.461942 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058166 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058170 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058175 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058178 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058181 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058183 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058192 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058195 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058198 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058200 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058204 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058207 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058210 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058213 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058218 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058221 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058223 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058227 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058229 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:24:01.462822 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058232 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058235 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058237 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058240 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058242 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058244 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058247 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058249 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058252 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058254 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058257 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058259 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058262 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058265 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058267 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058271 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058273 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058276 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058278 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058281 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:24:01.463686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058290 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058293 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058295 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058298 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058300 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058303 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058305 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058309 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058311 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058314 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058316 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058319 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058321 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058324 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058326 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058329 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058331 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058333 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058336 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058338 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:24:01.464573 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058341 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058343 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058346 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058348 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058351 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058353 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058357 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058361 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058365 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058378 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058382 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058385 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058388 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058397 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058401 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058403 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058406 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:24:01.465444 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:01.058409 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.058413 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.058535 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.060704 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.061948 2582 server.go:1019] "Starting client certificate rotation" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.062056 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.062092 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.065015 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.066534 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.076484 2582 log.go:25] "Validated CRI v1 runtime API" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.078668 2582 log.go:25] "Validated CRI v1 image API" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.079427 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 12:24:01.466325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.081603 2582 fs.go:135] Filesystem UUIDs: map[274f3378-b6e3-4fd9-a3ed-f46cf72ffc6e:/dev/nvme0n1p3 5e8c991e-5c67-4b08-aa1a-99c93e0a55ef:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 15 12:24:01.466976 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.081622 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.085324 2582 manager.go:216] Machine: {Timestamp:2026-07-15 12:24:01.084855322 +0000 UTC m=+0.297601823 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104776 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2072b4dc105623f3395fa3cc6b55f9 SystemUUID:ec2072b4-dc10-5623-f339-5fa3cc6b55f9 BootID:02d05ff1-7ea5-45db-9535-8b0a18a26c61 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:e4:0a:26:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:e4:0a:26:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:13:3b:02:e4:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.085420 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.085487 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.086976 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.086999 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-235.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.087721 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.087731 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 12:24:01.467185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.087738 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.088382 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.089803 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094200 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094309 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094711 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094721 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094732 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094741 2582 kubelet.go:398] "Adding apiserver pod source" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.094749 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.095116 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.095132 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.097074 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099461 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099488 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099663 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099682 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099689 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099694 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099701 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.099706 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100159 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100167 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100173 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100178 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100191 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100199 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100860 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.100868 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.103873 2582 server.go:1268] "Started kubelet" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.103942 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.103985 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.103962 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.108831 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.111988 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.112064 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.112309 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.112362 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 12:24:01.467828 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.112428 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-235.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.112474 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-235.ec2.internal.18c276166e4a1976 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-235.ec2.internal,UID:ip-10-0-128-235.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-235.ec2.internal,},FirstTimestamp:2026-07-15 12:24:01.10384575 +0000 UTC m=+0.316592253,LastTimestamp:2026-07-15 12:24:01.10384575 +0000 UTC m=+0.316592253,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-235.ec2.internal,}" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.114074 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.117590 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.117680 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.117663 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118091 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118159 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118176 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118296 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118306 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118475 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118487 2582 factory.go:55] Registering systemd factory Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118496 2582 factory.go:223] Registration of the systemd container factory successfully Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118685 2582 factory.go:153] Registering CRI-O factory Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118698 2582 factory.go:223] Registration of the crio container factory successfully Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118721 2582 factory.go:103] Registering Raw factory Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.118734 2582 manager.go:1196] Started watching for new ooms in manager Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.119048 2582 manager.go:349] Starting recovery of all containers Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.122875 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.122945 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.126635 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.128380 2582 manager.go:354] Recovery completed Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.134711 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.136596 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.136623 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.136633 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.137140 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.137150 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.137152 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.137171 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.137971 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.138002 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.469346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.138012 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.139400 2582 policy_none.go:49] "None policy: Start" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.139417 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.139427 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.141448 2582 policy_none.go:47] "Start" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.150092 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tcncf" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.151103 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-235.ec2.internal.18c27616703e0793 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-235.ec2.internal,UID:ip-10-0-128-235.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-235.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-235.ec2.internal,},FirstTimestamp:2026-07-15 12:24:01.136609171 +0000 UTC m=+0.349355673,LastTimestamp:2026-07-15 12:24:01.136609171 +0000 UTC m=+0.349355673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-235.ec2.internal,}" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.160835 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-235.ec2.internal.18c27616703e5126 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-235.ec2.internal,UID:ip-10-0-128-235.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-235.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-235.ec2.internal,},FirstTimestamp:2026-07-15 12:24:01.136628006 +0000 UTC m=+0.349374507,LastTimestamp:2026-07-15 12:24:01.136628006 +0000 UTC m=+0.349374507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-235.ec2.internal,}" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.164391 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tcncf" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.170984 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-235.ec2.internal.18c27616703e73be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-235.ec2.internal,UID:ip-10-0-128-235.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-235.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-235.ec2.internal,},FirstTimestamp:2026-07-15 12:24:01.136636862 +0000 UTC m=+0.349383364,LastTimestamp:2026-07-15 12:24:01.136636862 +0000 UTC m=+0.349383364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-235.ec2.internal,}" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206156 2582 manager.go:337] "Starting Device Plugin manager" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.206224 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206239 2582 server.go:87] "Starting device plugin registration server" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206487 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206497 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206591 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206668 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.206676 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.207336 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.207397 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.470883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.306859 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.307886 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.307914 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.307925 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.307953 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.308196 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.308230 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.308247 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.308282 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.311771 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.318933 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.318950 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-235.ec2.internal\": node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.408932 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal"] Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.409004 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.410433 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.410458 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.410468 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.412698 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.412816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.412843 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.413365 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.413370 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.413390 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.413389 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.413400 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.413408 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.416022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.416052 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.416705 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.416727 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.416739 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:24:01.474036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.423462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50acb58cc5a4f383f20dc1f8601eda81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"50acb58cc5a4f383f20dc1f8601eda81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.475544 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.423483 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50acb58cc5a4f383f20dc1f8601eda81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"50acb58cc5a4f383f20dc1f8601eda81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.475544 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.423501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/704cb184d5c204a08f126daf6f37229b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-235.ec2.internal\" (UID: \"704cb184d5c204a08f126daf6f37229b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.475544 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.440269 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-235.ec2.internal\" not found" node="ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.475544 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.444668 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-235.ec2.internal\" not found" node="ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.475544 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.458699 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.523788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50acb58cc5a4f383f20dc1f8601eda81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"50acb58cc5a4f383f20dc1f8601eda81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.523810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50acb58cc5a4f383f20dc1f8601eda81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"50acb58cc5a4f383f20dc1f8601eda81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.523826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/704cb184d5c204a08f126daf6f37229b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-235.ec2.internal\" (UID: \"704cb184d5c204a08f126daf6f37229b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.523870 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/704cb184d5c204a08f126daf6f37229b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-235.ec2.internal\" (UID: \"704cb184d5c204a08f126daf6f37229b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.523884 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50acb58cc5a4f383f20dc1f8601eda81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"50acb58cc5a4f383f20dc1f8601eda81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.523895 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50acb58cc5a4f383f20dc1f8601eda81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"50acb58cc5a4f383f20dc1f8601eda81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.560222 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.718528 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.661132 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.977621 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.743803 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.977621 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:01.750246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Jul 15 12:24:01.977621 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.762119 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.977621 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.864538 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:01.977621 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:01.965669 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:02.222681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.063344 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 12:24:02.222681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.063488 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:24:02.222681 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:02.066609 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:02.222681 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:02.167484 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Jul 15 12:24:02.222681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.168658 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 12:19:01 +0000 UTC" deadline="2028-03-20 06:57:58.232761163 +0000 UTC" Jul 15 12:24:02.222681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.168694 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14730h33m56.06407205s" Jul 15 12:24:02.443436 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.239336 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:24:02.443436 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.318783 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Jul 15 12:24:02.443436 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.343148 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 12:24:02.443436 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.344803 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Jul 15 12:24:02.443436 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.357592 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 12:24:02.646119 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.498471 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:24:02.646119 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:02.532633 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704cb184d5c204a08f126daf6f37229b.slice/crio-570827df308d7099b4264c0dfd974c197551ee510ee513320caa627833cf6111 WatchSource:0}: Error finding container 570827df308d7099b4264c0dfd974c197551ee510ee513320caa627833cf6111: Status 404 returned error can't find the container with id 570827df308d7099b4264c0dfd974c197551ee510ee513320caa627833cf6111 Jul 15 12:24:02.646119 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:02.533790 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50acb58cc5a4f383f20dc1f8601eda81.slice/crio-2c7412e3e710a857e5dfa2d7c3f8dde0af5de6bb34134b3cf40b36afb23fe262 WatchSource:0}: Error finding container 2c7412e3e710a857e5dfa2d7c3f8dde0af5de6bb34134b3cf40b36afb23fe262: Status 404 returned error can't find the container with id 2c7412e3e710a857e5dfa2d7c3f8dde0af5de6bb34134b3cf40b36afb23fe262 Jul 15 12:24:02.646119 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.539035 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:24:02.787874 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:02.718649 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.057449 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.097148 2582 apiserver.go:52] "Watching apiserver" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.109940 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.110474 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-tvnkk","openshift-network-operator/iptables-alerter-vvhj8","kube-system/global-pull-secret-syncer-x9hqp","kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52","openshift-cluster-node-tuning-operator/tuned-h24k5","openshift-image-registry/node-ca-58gtp","openshift-multus/multus-additional-cni-plugins-rpr5r","openshift-multus/network-metrics-daemon-gvnfn","openshift-ovn-kubernetes/ovnkube-node-t6qvz","kube-system/konnectivity-agent-n4qp6","openshift-dns/node-resolver-v2h2z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal","openshift-multus/multus-rcsb9"] Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.113383 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.115949 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.120790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.120897 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.120995 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.121032 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.121106 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bbq2d\"" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-netns\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-hostroot\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-conf-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-multus-certs\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-etc-kubernetes\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.166911 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-env-overrides\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-kubelet\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131881 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-log-socket\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ace04528-3fd0-4707-9668-19ce41a4d1e4-konnectivity-ca\") pod \"konnectivity-agent-n4qp6\" (UID: \"ace04528-3fd0-4707-9668-19ce41a4d1e4\") " pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.131992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-os-release\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-cni-bin\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-node-log\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-slash\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovnkube-script-lib\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d23692c8-7189-494b-aa94-15feeeda6119-dbus\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-cni-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-cni-binary-copy\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.170778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-var-lib-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132344 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-system-cni-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-cni-multus\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdpxn\" (UniqueName: \"kubernetes.io/projected/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-kube-api-access-bdpxn\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-systemd-units\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-cni-netd\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132539 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52m2r\" (UniqueName: \"kubernetes.io/projected/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-kube-api-access-52m2r\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-kubelet\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.132631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-systemd\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-cni-bin\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovn-node-metrics-cert\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ace04528-3fd0-4707-9668-19ce41a4d1e4-agent-certs\") pod \"konnectivity-agent-n4qp6\" (UID: \"ace04528-3fd0-4707-9668-19ce41a4d1e4\") " pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-daemon-config\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-run-netns\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.171811 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-ovn\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-cnibin\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-socket-dir-parent\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-k8s-cni-cncf-io\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-etc-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovnkube-config\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.133666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d23692c8-7189-494b-aa94-15feeeda6119-kubelet-config\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.137232 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8f6cc\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.137330 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.137690 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.137884 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.137921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.138036 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.138057 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.138240 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.139688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.142248 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.142662 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.142705 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z4tt6\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.142731 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.142904 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.143299 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.144837 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.174841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.145808 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.146265 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.146645 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.146645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-knk7r\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.147358 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.147442 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.148317 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.148416 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.148589 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vchks\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.149913 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.152390 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.152868 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.152899 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m6j6l\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.152918 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.153245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.155768 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.155832 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.155998 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.156615 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.156736 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.156923 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ssjzw\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.158315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.160927 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.162657 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.162679 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.162732 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.162980 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q8cjk\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.166171 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.176179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.166459 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.181050 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.166493 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pgv42\"" Jul 15 12:24:03.186504 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.184286 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rjpr" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.197536 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rjpr" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.219493 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.233872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.233901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-kubernetes\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.233936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-log-socket\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.233965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ace04528-3fd0-4707-9668-19ce41a4d1e4-konnectivity-ca\") pod \"konnectivity-agent-n4qp6\" (UID: \"ace04528-3fd0-4707-9668-19ce41a4d1e4\") " pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.233990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-os-release\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysconfig\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/741c47d3-0958-4711-b8af-0284dc0914de-etc-tuned\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234066 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-log-socket\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hk8\" (UniqueName: \"kubernetes.io/projected/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-kube-api-access-h4hk8\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234129 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-os-release\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234132 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-host\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3df168ff-f781-4feb-a4df-f491a68c3f38-hosts-file\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-cni-binary-copy\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.306599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-var-lib-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdpxn\" (UniqueName: \"kubernetes.io/projected/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-kube-api-access-bdpxn\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-registration-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-sys-fs\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-run\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234561 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-var-lib-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52m2r\" (UniqueName: \"kubernetes.io/projected/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-kube-api-access-52m2r\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysctl-conf\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-systemd\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234838 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ace04528-3fd0-4707-9668-19ce41a4d1e4-konnectivity-ca\") pod \"konnectivity-agent-n4qp6\" (UID: \"ace04528-3fd0-4707-9668-19ce41a4d1e4\") " pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovn-node-metrics-cert\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.314572 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234869 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-cni-binary-copy\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-daemon-config\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-systemd\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-lib-modules\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.234983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gt4\" (UniqueName: \"kubernetes.io/projected/8054ba69-f69c-4552-b97b-337a0659fd90-kube-api-access-d6gt4\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-cnibin\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-socket-dir-parent\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-k8s-cni-cncf-io\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235123 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-k8s-cni-cncf-io\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-netns\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-netns\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-conf-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235201 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 12:24:03.315496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235251 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-cnibin\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-conf-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-etc-kubernetes\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-etc-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-etc-kubernetes\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovnkube-config\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235285 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-socket-dir-parent\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d23692c8-7189-494b-aa94-15feeeda6119-kubelet-config\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235369 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-etc-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-multus-certs\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-daemon-config\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235397 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-run-multus-certs\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-modprobe-d\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-device-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-sys\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.437364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235461 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d23692c8-7189-494b-aa94-15feeeda6119-kubelet-config\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-host\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-kubelet\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741c47d3-0958-4711-b8af-0284dc0914de-tmp\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cnibin\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-openvswitch\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235663 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-kubelet\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8054ba69-f69c-4552-b97b-337a0659fd90-iptables-alerter-script\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-serviceca\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-cni-bin\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.438722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235836 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-cni-bin\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235899 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovnkube-config\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235908 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df168ff-f781-4feb-a4df-f491a68c3f38-tmp-dir\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.235920 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njwh\" (UniqueName: \"kubernetes.io/projected/3df168ff-f781-4feb-a4df-f491a68c3f38-kube-api-access-9njwh\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.235971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-node-log\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.236006 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:24:03.735983082 +0000 UTC m=+2.948729573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236015 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-node-log\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236024 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-slash\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovnkube-script-lib\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d23692c8-7189-494b-aa94-15feeeda6119-dbus\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-cni-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-slash\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.440620 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-system-cni-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-cni-multus\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d23692c8-7189-494b-aa94-15feeeda6119-dbus\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236273 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-host-var-lib-cni-multus\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-systemd-units\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236284 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-system-cni-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-multus-cni-dir\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236313 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-systemd-units\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-cni-netd\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-cni-netd\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-socket-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-systemd\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlmv\" (UniqueName: \"kubernetes.io/projected/741c47d3-0958-4711-b8af-0284dc0914de-kube-api-access-hrlmv\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsf6\" (UniqueName: \"kubernetes.io/projected/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-kube-api-access-gbsf6\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.442928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-kubelet\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-cni-bin\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236524 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ace04528-3fd0-4707-9668-19ce41a4d1e4-agent-certs\") pod \"konnectivity-agent-n4qp6\" (UID: \"ace04528-3fd0-4707-9668-19ce41a4d1e4\") " pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovnkube-script-lib\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-kubelet\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82bg\" (UniqueName: \"kubernetes.io/projected/30f8f986-f13d-4ba8-b62d-44316a98c576-kube-api-access-g82bg\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-os-release\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-cni-bin\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-run-netns\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-ovn\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-run-netns\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8054ba69-f69c-4552-b97b-337a0659fd90-host-slash\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-hostroot\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-hostroot\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysctl-d\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.443908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-run-ovn\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-var-lib-kubelet\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-system-cni-dir\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.236994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.237038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hwm\" (UniqueName: \"kubernetes.io/projected/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-kube-api-access-j8hwm\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.237068 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.237095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-env-overrides\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.237121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.237523 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-env-overrides\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.239770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-ovn-node-metrics-cert\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.240026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ace04528-3fd0-4707-9668-19ce41a4d1e4-agent-certs\") pod \"konnectivity-agent-n4qp6\" (UID: \"ace04528-3fd0-4707-9668-19ce41a4d1e4\") " pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.247318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52m2r\" (UniqueName: \"kubernetes.io/projected/e06cd6ee-a6f0-4ab1-95e7-f9154551fac2-kube-api-access-52m2r\") pod \"ovnkube-node-t6qvz\" (UID: \"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.247735 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdpxn\" (UniqueName: \"kubernetes.io/projected/3abd8f3f-2f2c-4847-8f51-7cf6c6706108-kube-api-access-bdpxn\") pod \"multus-rcsb9\" (UID: \"3abd8f3f-2f2c-4847-8f51-7cf6c6706108\") " pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.313923 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" event={"ID":"704cb184d5c204a08f126daf6f37229b","Type":"ContainerStarted","Data":"570827df308d7099b4264c0dfd974c197551ee510ee513320caa627833cf6111"} Jul 15 12:24:03.446681 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.315261 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" event={"ID":"50acb58cc5a4f383f20dc1f8601eda81","Type":"ContainerStarted","Data":"2c7412e3e710a857e5dfa2d7c3f8dde0af5de6bb34134b3cf40b36afb23fe262"} Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysctl-d\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-var-lib-kubelet\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-system-cni-dir\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hwm\" (UniqueName: \"kubernetes.io/projected/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-kube-api-access-j8hwm\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338522 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-var-lib-kubelet\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338549 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-system-cni-dir\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-kubernetes\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysctl-d\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338681 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-kubernetes\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysconfig\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.448095 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/741c47d3-0958-4711-b8af-0284dc0914de-etc-tuned\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338751 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysconfig\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hk8\" (UniqueName: \"kubernetes.io/projected/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-kube-api-access-h4hk8\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-host\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3df168ff-f781-4feb-a4df-f491a68c3f38-hosts-file\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-registration-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338897 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-host\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-sys-fs\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338950 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-sys-fs\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-run\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.338997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-run\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.449476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysctl-conf\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339086 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-registration-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339152 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3df168ff-f781-4feb-a4df-f491a68c3f38-hosts-file\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-lib-modules\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gt4\" (UniqueName: \"kubernetes.io/projected/8054ba69-f69c-4552-b97b-337a0659fd90-kube-api-access-d6gt4\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-sysctl-conf\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.339285 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-modprobe-d\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.339350 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:24:03.8393307 +0000 UTC m=+3.052077192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-device-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-sys\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-device-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.452334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-lib-modules\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-modprobe-d\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339471 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-host\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339483 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-sys\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-host\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741c47d3-0958-4711-b8af-0284dc0914de-tmp\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cnibin\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8054ba69-f69c-4552-b97b-337a0659fd90-iptables-alerter-script\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339666 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cnibin\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.339968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-serviceca\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df168ff-f781-4feb-a4df-f491a68c3f38-tmp-dir\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8054ba69-f69c-4552-b97b-337a0659fd90-iptables-alerter-script\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njwh\" (UniqueName: \"kubernetes.io/projected/3df168ff-f781-4feb-a4df-f491a68c3f38-kube-api-access-9njwh\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.453495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df168ff-f781-4feb-a4df-f491a68c3f38-tmp-dir\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-socket-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-systemd\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlmv\" (UniqueName: \"kubernetes.io/projected/741c47d3-0958-4711-b8af-0284dc0914de-kube-api-access-hrlmv\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsf6\" (UniqueName: \"kubernetes.io/projected/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-kube-api-access-gbsf6\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82bg\" (UniqueName: \"kubernetes.io/projected/30f8f986-f13d-4ba8-b62d-44316a98c576-kube-api-access-g82bg\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-serviceca\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-os-release\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8054ba69-f69c-4552-b97b-337a0659fd90-host-slash\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8054ba69-f69c-4552-b97b-337a0659fd90-host-slash\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340838 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-os-release\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340942 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/741c47d3-0958-4711-b8af-0284dc0914de-etc-systemd\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.340960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30f8f986-f13d-4ba8-b62d-44316a98c576-socket-dir\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.454223 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.341061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.341273 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.341609 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.341764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/741c47d3-0958-4711-b8af-0284dc0914de-etc-tuned\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.342662 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741c47d3-0958-4711-b8af-0284dc0914de-tmp\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.356447 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.356473 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.356491 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.356590 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:24:03.856542744 +0000 UTC m=+3.069289249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.362568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlmv\" (UniqueName: \"kubernetes.io/projected/741c47d3-0958-4711-b8af-0284dc0914de-kube-api-access-hrlmv\") pod \"tuned-h24k5\" (UID: \"741c47d3-0958-4711-b8af-0284dc0914de\") " pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.365651 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njwh\" (UniqueName: \"kubernetes.io/projected/3df168ff-f781-4feb-a4df-f491a68c3f38-kube-api-access-9njwh\") pod \"node-resolver-v2h2z\" (UID: \"3df168ff-f781-4feb-a4df-f491a68c3f38\") " pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.365819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82bg\" (UniqueName: \"kubernetes.io/projected/30f8f986-f13d-4ba8-b62d-44316a98c576-kube-api-access-g82bg\") pod \"aws-ebs-csi-driver-node-c6k52\" (UID: \"30f8f986-f13d-4ba8-b62d-44316a98c576\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.365852 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsf6\" (UniqueName: \"kubernetes.io/projected/c4e87e0c-8851-4a19-8a10-25e541e5d6f4-kube-api-access-gbsf6\") pod \"multus-additional-cni-plugins-rpr5r\" (UID: \"c4e87e0c-8851-4a19-8a10-25e541e5d6f4\") " pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.366246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hwm\" (UniqueName: \"kubernetes.io/projected/cc03c29a-948a-4119-bb98-f4eb2d8ca13c-kube-api-access-j8hwm\") pod \"node-ca-58gtp\" (UID: \"cc03c29a-948a-4119-bb98-f4eb2d8ca13c\") " pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.454936 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.368211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gt4\" (UniqueName: \"kubernetes.io/projected/8054ba69-f69c-4552-b97b-337a0659fd90-kube-api-access-d6gt4\") pod \"iptables-alerter-vvhj8\" (UID: \"8054ba69-f69c-4552-b97b-337a0659fd90\") " pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.455584 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.368451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hk8\" (UniqueName: \"kubernetes.io/projected/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-kube-api-access-h4hk8\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.455584 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.446705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:03.456413 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.456382 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:03.461189 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.461071 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace04528_3fd0_4707_9668_19ce41a4d1e4.slice/crio-e460c7564d6c1736ced66236c84957a93557bdb477ceb8472ea59bbd1e42ae26 WatchSource:0}: Error finding container e460c7564d6c1736ced66236c84957a93557bdb477ceb8472ea59bbd1e42ae26: Status 404 returned error can't find the container with id e460c7564d6c1736ced66236c84957a93557bdb477ceb8472ea59bbd1e42ae26 Jul 15 12:24:03.467046 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.467016 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rcsb9" Jul 15 12:24:03.468259 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.468227 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06cd6ee_a6f0_4ab1_95e7_f9154551fac2.slice/crio-ac0b2c13694aef00de2f4d80ce2b3080c95b46a071f8ea8c5adb86fa6ca4c1e0 WatchSource:0}: Error finding container ac0b2c13694aef00de2f4d80ce2b3080c95b46a071f8ea8c5adb86fa6ca4c1e0: Status 404 returned error can't find the container with id ac0b2c13694aef00de2f4d80ce2b3080c95b46a071f8ea8c5adb86fa6ca4c1e0 Jul 15 12:24:03.472831 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.472754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" Jul 15 12:24:03.481803 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.481399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h24k5" Jul 15 12:24:03.482009 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.481978 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abd8f3f_2f2c_4847_8f51_7cf6c6706108.slice/crio-69a11045bd3709ccd62bbe006e1a2a27b12a0d9ece5f2bb6200d0562369c7913 WatchSource:0}: Error finding container 69a11045bd3709ccd62bbe006e1a2a27b12a0d9ece5f2bb6200d0562369c7913: Status 404 returned error can't find the container with id 69a11045bd3709ccd62bbe006e1a2a27b12a0d9ece5f2bb6200d0562369c7913 Jul 15 12:24:03.486169 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.486143 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f8f986_f13d_4ba8_b62d_44316a98c576.slice/crio-81c69b71d182ed727ef196b63d776c3050e5d9ad0efe63b561ec07c273b920c5 WatchSource:0}: Error finding container 81c69b71d182ed727ef196b63d776c3050e5d9ad0efe63b561ec07c273b920c5: Status 404 returned error can't find the container with id 81c69b71d182ed727ef196b63d776c3050e5d9ad0efe63b561ec07c273b920c5 Jul 15 12:24:03.488942 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.488771 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" Jul 15 12:24:03.493072 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.493048 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741c47d3_0958_4711_b8af_0284dc0914de.slice/crio-5ce3034ec21ebcd4f324933b7609369f008c3d932a0ab13c8aeb98c7b33fa387 WatchSource:0}: Error finding container 5ce3034ec21ebcd4f324933b7609369f008c3d932a0ab13c8aeb98c7b33fa387: Status 404 returned error can't find the container with id 5ce3034ec21ebcd4f324933b7609369f008c3d932a0ab13c8aeb98c7b33fa387 Jul 15 12:24:03.500345 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.500320 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e87e0c_8851_4a19_8a10_25e541e5d6f4.slice/crio-028f038e312496fb374d73835da5016612f85afc55f6df3479684c757787864c WatchSource:0}: Error finding container 028f038e312496fb374d73835da5016612f85afc55f6df3479684c757787864c: Status 404 returned error can't find the container with id 028f038e312496fb374d73835da5016612f85afc55f6df3479684c757787864c Jul 15 12:24:03.532967 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.532941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vvhj8" Jul 15 12:24:03.543492 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.543467 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054ba69_f69c_4552_b97b_337a0659fd90.slice/crio-64c85bc4ffdf1369ec71f0a1be7de1b8bfed279e0037cefeca992834b36787e9 WatchSource:0}: Error finding container 64c85bc4ffdf1369ec71f0a1be7de1b8bfed279e0037cefeca992834b36787e9: Status 404 returned error can't find the container with id 64c85bc4ffdf1369ec71f0a1be7de1b8bfed279e0037cefeca992834b36787e9 Jul 15 12:24:03.573693 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.573671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58gtp" Jul 15 12:24:03.578792 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.578773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2h2z" Jul 15 12:24:03.742498 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.742458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:03.742658 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.742619 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:03.742707 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.742691 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:24:04.742675202 +0000 UTC m=+3.955421696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:03.843450 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.843359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:03.843636 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.843510 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:03.843636 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.843570 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:24:04.843552598 +0000 UTC m=+4.056299112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:03.894937 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.894910 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc03c29a_948a_4119_bb98_f4eb2d8ca13c.slice/crio-7fc97f5be7a0256823f57434100c82878a77d642f13586546b9b4815dc9d299c WatchSource:0}: Error finding container 7fc97f5be7a0256823f57434100c82878a77d642f13586546b9b4815dc9d299c: Status 404 returned error can't find the container with id 7fc97f5be7a0256823f57434100c82878a77d642f13586546b9b4815dc9d299c Jul 15 12:24:03.895865 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:03.895834 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df168ff_f781_4feb_a4df_f491a68c3f38.slice/crio-69b66fa6badc08452c0cc735d42bc0fac46e31d7fcbb25a0aca8d1c6be70964b WatchSource:0}: Error finding container 69b66fa6badc08452c0cc735d42bc0fac46e31d7fcbb25a0aca8d1c6be70964b: Status 404 returned error can't find the container with id 69b66fa6badc08452c0cc735d42bc0fac46e31d7fcbb25a0aca8d1c6be70964b Jul 15 12:24:03.943718 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:03.943687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:03.943849 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.943832 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:03.943968 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.943852 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:03.943968 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.943863 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:03.943968 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:03.943921 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:24:04.943901497 +0000 UTC m=+4.156648006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:04.201066 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.200887 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 12:19:03 +0000 UTC" deadline="2028-02-07 03:37:42.851519766 +0000 UTC" Jul 15 12:24:04.201066 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.200952 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13719h13m38.650572225s" Jul 15 12:24:04.309455 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.309244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:04.309455 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.309385 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:04.340734 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.340699 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2h2z" event={"ID":"3df168ff-f781-4feb-a4df-f491a68c3f38","Type":"ContainerStarted","Data":"69b66fa6badc08452c0cc735d42bc0fac46e31d7fcbb25a0aca8d1c6be70964b"} Jul 15 12:24:04.342784 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.342681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58gtp" event={"ID":"cc03c29a-948a-4119-bb98-f4eb2d8ca13c","Type":"ContainerStarted","Data":"7fc97f5be7a0256823f57434100c82878a77d642f13586546b9b4815dc9d299c"} Jul 15 12:24:04.347261 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.347180 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vvhj8" event={"ID":"8054ba69-f69c-4552-b97b-337a0659fd90","Type":"ContainerStarted","Data":"64c85bc4ffdf1369ec71f0a1be7de1b8bfed279e0037cefeca992834b36787e9"} Jul 15 12:24:04.356955 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.356876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h24k5" event={"ID":"741c47d3-0958-4711-b8af-0284dc0914de","Type":"ContainerStarted","Data":"5ce3034ec21ebcd4f324933b7609369f008c3d932a0ab13c8aeb98c7b33fa387"} Jul 15 12:24:04.360645 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.360619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rcsb9" event={"ID":"3abd8f3f-2f2c-4847-8f51-7cf6c6706108","Type":"ContainerStarted","Data":"69a11045bd3709ccd62bbe006e1a2a27b12a0d9ece5f2bb6200d0562369c7913"} Jul 15 12:24:04.367980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.367953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"ac0b2c13694aef00de2f4d80ce2b3080c95b46a071f8ea8c5adb86fa6ca4c1e0"} Jul 15 12:24:04.373512 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.373487 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerStarted","Data":"028f038e312496fb374d73835da5016612f85afc55f6df3479684c757787864c"} Jul 15 12:24:04.387201 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.387173 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" event={"ID":"30f8f986-f13d-4ba8-b62d-44316a98c576","Type":"ContainerStarted","Data":"81c69b71d182ed727ef196b63d776c3050e5d9ad0efe63b561ec07c273b920c5"} Jul 15 12:24:04.394738 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.394711 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4qp6" event={"ID":"ace04528-3fd0-4707-9668-19ce41a4d1e4","Type":"ContainerStarted","Data":"e460c7564d6c1736ced66236c84957a93557bdb477ceb8472ea59bbd1e42ae26"} Jul 15 12:24:04.756660 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.756512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:04.756809 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.756692 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:04.756809 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.756764 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:24:06.756742956 +0000 UTC m=+5.969489449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:04.857286 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.857254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:04.857457 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.857395 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:04.857457 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.857453 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:24:06.857435443 +0000 UTC m=+6.070181939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:04.957686 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:04.957653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:04.957842 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.957807 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:04.957842 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.957824 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:04.957842 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.957837 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:04.958012 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:04.957890 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:24:06.957872086 +0000 UTC m=+6.170618579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:05.201712 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:05.201604 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 12:19:03 +0000 UTC" deadline="2028-04-05 08:45:19.835838892 +0000 UTC" Jul 15 12:24:05.201712 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:05.201664 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15116h21m14.634179095s" Jul 15 12:24:05.321666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:05.321639 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:05.321666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:05.321658 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:05.322142 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:05.321760 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:05.322142 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:05.321889 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:05.431622 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:05.430677 2582 generic.go:356] "Generic (PLEG): container finished" podID="50acb58cc5a4f383f20dc1f8601eda81" containerID="cbe8f6c0693114b776e34a1428d1787fc7fb734eafc59a1481d29bf85070e97e" exitCode=0 Jul 15 12:24:05.431622 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:05.430725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" event={"ID":"50acb58cc5a4f383f20dc1f8601eda81","Type":"ContainerDied","Data":"cbe8f6c0693114b776e34a1428d1787fc7fb734eafc59a1481d29bf85070e97e"} Jul 15 12:24:06.309130 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:06.309096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:06.309297 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.309236 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:06.450623 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:06.450067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" event={"ID":"50acb58cc5a4f383f20dc1f8601eda81","Type":"ContainerStarted","Data":"5d6491fade0106fbc58c528dc204e06f64bdc1b5909d8a83a5b0cbf238d1b36c"} Jul 15 12:24:06.770830 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:06.770749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:06.771070 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.770907 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:06.771070 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.770972 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:24:10.770954179 +0000 UTC m=+9.983700670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:06.872448 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:06.871870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:06.872448 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.872017 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:06.872448 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.872079 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:24:10.872059256 +0000 UTC m=+10.084805747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:06.973199 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:06.973157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:06.973416 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.973368 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:06.973416 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.973393 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:06.973416 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.973407 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:06.973635 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:06.973468 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:24:10.973449544 +0000 UTC m=+10.186196046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:07.312508 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:07.312478 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:07.312694 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:07.312633 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:07.313013 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:07.312991 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:07.313137 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:07.313094 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:08.309396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:08.309359 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:08.309880 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:08.309499 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:09.309172 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:09.309137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:09.309343 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:09.309286 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:09.309423 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:09.309335 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:09.309797 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:09.309447 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:10.308983 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:10.308945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:10.309173 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:10.309083 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:10.806202 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:10.806113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:10.806640 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:10.806278 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:10.806640 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:10.806354 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:24:18.806334695 +0000 UTC m=+18.019081197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:10.906503 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:10.906477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:10.906672 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:10.906613 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:10.906672 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:10.906657 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:24:18.906645035 +0000 UTC m=+18.119391524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:11.007847 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:11.007775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:11.008033 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:11.007951 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:11.008033 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:11.007969 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:11.008033 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:11.007984 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:11.008196 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:11.008047 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:24:19.008028358 +0000 UTC m=+18.220774851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:11.309885 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:11.309653 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:11.309885 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:11.309775 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:11.309885 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:11.309783 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:11.309885 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:11.309858 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:12.309513 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:12.309471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:12.309965 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:12.309619 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:13.312190 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:13.312155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:13.312190 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:13.312184 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:13.312664 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:13.312288 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:13.312664 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:13.312418 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:14.309370 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:14.309337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:14.309529 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:14.309446 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:15.309011 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:15.308923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:15.309447 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:15.309061 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:15.309447 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:15.309118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:15.309447 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:15.309230 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:16.308850 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:16.308805 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:16.309004 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:16.308934 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:17.308535 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:17.308500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:17.308971 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:17.308657 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:17.308971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:17.308694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:17.308971 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:17.308770 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:18.308908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:18.308868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:18.309401 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:18.309005 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:18.867240 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:18.867206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:18.867402 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:18.867360 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:18.867446 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:18.867421 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:24:34.867406587 +0000 UTC m=+34.080153080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:18.968332 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:18.968293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:18.968513 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:18.968467 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:18.968604 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:18.968544 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:24:34.968521582 +0000 UTC m=+34.181268086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:19.069138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:19.069101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:19.069284 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:19.069236 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:19.069284 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:19.069251 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:19.069284 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:19.069261 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:19.069420 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:19.069317 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:24:35.069300116 +0000 UTC m=+34.282046610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:19.308919 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:19.308834 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:19.309324 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:19.308970 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:19.309324 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:19.309017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:19.309324 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:19.309104 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:20.309348 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:20.309311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:20.309867 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:20.309438 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:21.311757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:21.311728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:21.312200 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:21.311731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:21.312200 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:21.311836 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:21.312200 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:21.311964 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:22.309259 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:22.309221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:22.309424 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:22.309338 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:23.311461 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.311333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:23.311871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.311364 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:23.311871 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:23.311537 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:23.311871 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:23.311712 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:23.484321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.484289 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rcsb9" event={"ID":"3abd8f3f-2f2c-4847-8f51-7cf6c6706108","Type":"ContainerStarted","Data":"40b6348b13df30b5840e396eaa4d7a3f8ba23152a5339a19febf98b3a31d0109"} Jul 15 12:24:23.486093 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.486072 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:24:23.486473 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.486441 2582 generic.go:356] "Generic (PLEG): container finished" podID="e06cd6ee-a6f0-4ab1-95e7-f9154551fac2" containerID="41b78cce92ad7d09f9a3fe99664118e5c2d4d7c0c900c2e66c0439102a897481" exitCode=1 Jul 15 12:24:23.486548 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.486510 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerDied","Data":"41b78cce92ad7d09f9a3fe99664118e5c2d4d7c0c900c2e66c0439102a897481"} Jul 15 12:24:23.486639 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.486619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"23bd94c64cb4c9dc56cfe8cbfc208d72d306b06278bb001719cc2dfade26ca8d"} Jul 15 12:24:23.488218 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.488195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerStarted","Data":"74c1b595eabc5ebb21f5c78ebbd93e677aa47e3a9c702c9d453547d61ff1b6d1"} Jul 15 12:24:23.490047 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.489958 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" event={"ID":"30f8f986-f13d-4ba8-b62d-44316a98c576","Type":"ContainerStarted","Data":"9e6b141a0959f92d871364de9db7c650701cb60ab961d4f215c741ac84c3e3bf"} Jul 15 12:24:23.492866 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.492841 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4qp6" event={"ID":"ace04528-3fd0-4707-9668-19ce41a4d1e4","Type":"ContainerStarted","Data":"9d8ed013acacd8b9279c5a1a7e89fb3fcf965e3c2c5723389fcc544eb33f5e2f"} Jul 15 12:24:23.494121 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.494098 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" event={"ID":"704cb184d5c204a08f126daf6f37229b","Type":"ContainerStarted","Data":"08678450cd7511052493432343c8e97a282c7cd8df3eee6b47de80095002b737"} Jul 15 12:24:23.495554 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.495530 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2h2z" event={"ID":"3df168ff-f781-4feb-a4df-f491a68c3f38","Type":"ContainerStarted","Data":"55df3f1729b9668d20e326d7c8420cb944284ec75be14a135c84042b6f6bbfef"} Jul 15 12:24:23.497002 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.496979 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58gtp" event={"ID":"cc03c29a-948a-4119-bb98-f4eb2d8ca13c","Type":"ContainerStarted","Data":"35caba8da34cc8bbff6f45776f3a3d5b2f1aadff0d352f2dfc09a04f215d86ce"} Jul 15 12:24:23.498426 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.498407 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h24k5" event={"ID":"741c47d3-0958-4711-b8af-0284dc0914de","Type":"ContainerStarted","Data":"97378c71ee3d1dcc016a3648bcdee11ebeb41fa9f033d83f02c186eef86f6f6f"} Jul 15 12:24:23.506319 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.506281 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" podStartSLOduration=21.506270989 podStartE2EDuration="21.506270989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:06.483946986 +0000 UTC m=+5.696693498" watchObservedRunningTime="2026-07-15 12:24:23.506270989 +0000 UTC m=+22.719017500" Jul 15 12:24:23.506413 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.506352 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rcsb9" podStartSLOduration=2.828170331 podStartE2EDuration="22.506345585s" totalImagesPullingTime="19.678175254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:23.506198043 +0000 UTC m=+22.718944619" watchObservedRunningTime="2026-07-15 12:24:23.506345585 +0000 UTC m=+22.719092093" Jul 15 12:24:23.534664 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.534620 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n4qp6" podStartSLOduration=2.891589461 podStartE2EDuration="22.534606474s" totalImagesPullingTime="19.643017013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:23.528723385 +0000 UTC m=+22.741469899" watchObservedRunningTime="2026-07-15 12:24:23.534606474 +0000 UTC m=+22.747352985" Jul 15 12:24:23.565643 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.565580 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-58gtp" podStartSLOduration=3.355254263 podStartE2EDuration="22.565544877s" totalImagesPullingTime="19.210290614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:23.545710051 +0000 UTC m=+22.758456562" watchObservedRunningTime="2026-07-15 12:24:23.565544877 +0000 UTC m=+22.778291388" Jul 15 12:24:23.587376 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.587295 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" podStartSLOduration=21.587280727 podStartE2EDuration="21.587280727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:23.565537855 +0000 UTC m=+22.778284369" watchObservedRunningTime="2026-07-15 12:24:23.587280727 +0000 UTC m=+22.800027282" Jul 15 12:24:23.587522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.587408 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h24k5" podStartSLOduration=2.964661908 podStartE2EDuration="22.587401212s" totalImagesPullingTime="19.622739304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:23.586622969 +0000 UTC m=+22.799369479" watchObservedRunningTime="2026-07-15 12:24:23.587401212 +0000 UTC m=+22.800147726" Jul 15 12:24:23.641819 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.641777 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v2h2z" podStartSLOduration=3.432613517 podStartE2EDuration="22.641762856s" totalImagesPullingTime="19.209149339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:23.641538596 +0000 UTC m=+22.854285121" watchObservedRunningTime="2026-07-15 12:24:23.641762856 +0000 UTC m=+22.854509366" Jul 15 12:24:23.722927 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.722832 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:23.723418 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:23.723402 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:24.309205 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.309170 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:24.309348 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:24.309300 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:24.502169 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.502100 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vvhj8" event={"ID":"8054ba69-f69c-4552-b97b-337a0659fd90","Type":"ContainerStarted","Data":"b5be3b6d0ee7542a6dc8d8f2bd29988e42eb139617cee031bfb69e1b4c9bcd42"} Jul 15 12:24:24.504864 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.504839 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:24:24.505179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.505157 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"c36137c4a3e4471fcf4097113db4f1c0640b68efdaf89e9ff35907f06f1eaaaa"} Jul 15 12:24:24.505243 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.505228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"dd9a16a13056f97d848ecba52b2927f6304559ad90b1dcd2d8a726718b9b708d"} Jul 15 12:24:24.505284 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.505266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"a5d7d8a4a579f5e7ad4d847bf28c6dc201eed9b736a7cc0735ad24ae7d6a32be"} Jul 15 12:24:24.505322 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.505301 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"0f9d02cf1ac6e77dd7d57b5f3630a3fc928c29124380f21721a0c5d7941bba57"} Jul 15 12:24:24.506392 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.506371 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4e87e0c-8851-4a19-8a10-25e541e5d6f4" containerID="74c1b595eabc5ebb21f5c78ebbd93e677aa47e3a9c702c9d453547d61ff1b6d1" exitCode=0 Jul 15 12:24:24.506482 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.506450 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerDied","Data":"74c1b595eabc5ebb21f5c78ebbd93e677aa47e3a9c702c9d453547d61ff1b6d1"} Jul 15 12:24:24.507093 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.507076 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:24.507502 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.507486 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n4qp6" Jul 15 12:24:24.520732 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.520698 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vvhj8" podStartSLOduration=3.957434593 podStartE2EDuration="23.520688043s" totalImagesPullingTime="19.56325345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:24.520243015 +0000 UTC m=+23.732989527" watchObservedRunningTime="2026-07-15 12:24:24.520688043 +0000 UTC m=+23.733434554" Jul 15 12:24:24.677403 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:24.677383 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 12:24:25.221285 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:25.221169 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T12:24:24.677397919Z","UUID":"24e6fe75-fe28-4310-8204-ba1afffb7eef","Handler":null,"Name":"","Endpoint":""} Jul 15 12:24:25.223012 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:25.222994 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 12:24:25.223135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:25.223032 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 12:24:25.312313 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:25.312264 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:25.312454 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:25.312425 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:25.312506 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:25.312473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:25.312620 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:25.312600 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:25.511356 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:25.511276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" event={"ID":"30f8f986-f13d-4ba8-b62d-44316a98c576","Type":"ContainerStarted","Data":"b19c743287c47b22f0ec20ef3698c953321d790b7d466db72afd88842f760af5"} Jul 15 12:24:26.308971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:26.308699 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:26.309106 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:26.309023 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:26.516573 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:26.516529 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:24:26.517134 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:26.516904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"1944109e2b3e426e732a984b6e09f37881b7adeaa33a726a99341168696546f2"} Jul 15 12:24:26.519102 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:26.519069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" event={"ID":"30f8f986-f13d-4ba8-b62d-44316a98c576","Type":"ContainerStarted","Data":"16c8eff37cda4c0210682b2524f46225d8d927a148a466bc23b5ab4ec66ab4bd"} Jul 15 12:24:26.541181 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:26.541138 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k52" podStartSLOduration=3.042081503 podStartE2EDuration="25.541125351s" totalImagesPullingTime="22.499043848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:26.541062484 +0000 UTC m=+25.753808997" watchObservedRunningTime="2026-07-15 12:24:26.541125351 +0000 UTC m=+25.753871862" Jul 15 12:24:27.309417 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:27.309385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:27.309628 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:27.309385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:27.309628 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:27.309522 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:27.309628 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:27.309583 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:28.309224 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:28.309199 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:28.309731 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:28.309311 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:29.308894 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.308633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:29.309011 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.308699 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:29.309011 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:29.308979 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:29.309011 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:29.308997 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:29.528333 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.528311 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:24:29.528798 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.528645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"4d8fb6e13f982eba58ae6aed628af307b797ebf2937fa8a3f65bb6a8a99ce2bf"} Jul 15 12:24:29.528902 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.528877 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:29.529101 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.528997 2582 scope.go:117] "RemoveContainer" containerID="41b78cce92ad7d09f9a3fe99664118e5c2d4d7c0c900c2e66c0439102a897481" Jul 15 12:24:29.530651 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.530627 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4e87e0c-8851-4a19-8a10-25e541e5d6f4" containerID="1f9ba06964acb9ef203c137ff491ad78775ce4830af5fd12c3c69f915aa8389b" exitCode=0 Jul 15 12:24:29.530762 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.530677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerDied","Data":"1f9ba06964acb9ef203c137ff491ad78775ce4830af5fd12c3c69f915aa8389b"} Jul 15 12:24:29.544638 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.544619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:29.733277 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:29.733244 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:30.309226 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:30.309195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:30.309489 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:30.309294 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:30.535829 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:30.535808 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:24:30.536166 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:30.536144 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" event={"ID":"e06cd6ee-a6f0-4ab1-95e7-f9154551fac2","Type":"ContainerStarted","Data":"496fb5ec928461e147839663d1679903c8784c9efc15a16fef25090d902dbc36"} Jul 15 12:24:30.554245 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:30.554222 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:30.570943 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:30.570922 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:24:30.585679 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:30.585635 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" podStartSLOduration=9.870521854 podStartE2EDuration="29.585621317s" totalImagesPullingTime="19.715099463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:30.584822098 +0000 UTC m=+29.797568620" watchObservedRunningTime="2026-07-15 12:24:30.585621317 +0000 UTC m=+29.798367827" Jul 15 12:24:31.309171 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:31.309142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:31.309331 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:31.309235 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:31.309331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:31.309282 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:31.309401 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:31.309365 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:31.540460 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:31.540431 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4e87e0c-8851-4a19-8a10-25e541e5d6f4" containerID="bdb7f52e44643ececb747951163837560b9917d5cc63edf3f7d8163c46b4fdfa" exitCode=0 Jul 15 12:24:31.540891 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:31.540518 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerDied","Data":"bdb7f52e44643ececb747951163837560b9917d5cc63edf3f7d8163c46b4fdfa"} Jul 15 12:24:32.308987 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:32.308951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:32.309155 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:32.309083 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:33.311175 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:33.311147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:33.311511 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:33.311248 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:33.311511 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:33.311305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:33.311511 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:33.311381 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:33.546944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:33.546914 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4e87e0c-8851-4a19-8a10-25e541e5d6f4" containerID="928b9c894fd2721973b5394c1f780a2690ddb6a13031abe3aa7d3dc8ce917a2c" exitCode=0 Jul 15 12:24:33.547125 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:33.546958 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerDied","Data":"928b9c894fd2721973b5394c1f780a2690ddb6a13031abe3aa7d3dc8ce917a2c"} Jul 15 12:24:34.309138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.308972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:34.309341 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.309220 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:34.651286 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.651172 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvnkk"] Jul 15 12:24:34.651286 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.651282 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:34.651955 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.651380 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:34.655346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.655103 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9hqp"] Jul 15 12:24:34.655346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.655196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:34.655346 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.655284 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:34.656202 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.655875 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvnfn"] Jul 15 12:24:34.656202 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.655962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:34.656202 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.656056 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:34.890272 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.890191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:34.890437 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.890329 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:34.890437 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.890386 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret podName:d23692c8-7189-494b-aa94-15feeeda6119 nodeName:}" failed. No retries permitted until 2026-07-15 12:25:06.890368098 +0000 UTC m=+66.103114591 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret") pod "global-pull-secret-syncer-x9hqp" (UID: "d23692c8-7189-494b-aa94-15feeeda6119") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:24:34.990970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:34.990901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:34.991114 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.991058 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:34.991173 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:34.991126 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs podName:d1bac588-12e5-410d-8ea4-cb4ec9bbca1a nodeName:}" failed. No retries permitted until 2026-07-15 12:25:06.991107595 +0000 UTC m=+66.203854096 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs") pod "network-metrics-daemon-gvnfn" (UID: "d1bac588-12e5-410d-8ea4-cb4ec9bbca1a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:24:35.091889 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:35.091843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:35.092067 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:35.091967 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:24:35.092067 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:35.091988 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:24:35.092067 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:35.091998 2582 projected.go:196] Error preparing data for projected volume kube-api-access-rgtjx for pod openshift-network-diagnostics/network-check-target-tvnkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:35.092067 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:35.092051 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx podName:7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc nodeName:}" failed. No retries permitted until 2026-07-15 12:25:07.092033941 +0000 UTC m=+66.304780432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtjx" (UniqueName: "kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx") pod "network-check-target-tvnkk" (UID: "7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:24:36.309187 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:36.309152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:36.309809 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:36.309265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:36.309809 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:36.309289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:36.309809 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:36.309279 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:36.309809 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:36.309370 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:36.309809 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:36.309453 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:38.309022 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:38.308956 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:38.309022 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:38.308990 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:38.309022 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:38.308992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:38.309397 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:38.309071 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9hqp" podUID="d23692c8-7189-494b-aa94-15feeeda6119" Jul 15 12:24:38.309397 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:38.309187 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvnfn" podUID="d1bac588-12e5-410d-8ea4-cb4ec9bbca1a" Jul 15 12:24:38.309397 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:38.309266 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvnkk" podUID="7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc" Jul 15 12:24:38.563486 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:38.563421 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4e87e0c-8851-4a19-8a10-25e541e5d6f4" containerID="b482021366ebc32a3f07e55ca2f7915ef836229f115e96382a80fb192661c3cf" exitCode=0 Jul 15 12:24:38.563486 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:38.563458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerDied","Data":"b482021366ebc32a3f07e55ca2f7915ef836229f115e96382a80fb192661c3cf"} Jul 15 12:24:39.126988 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.126825 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeReady" Jul 15 12:24:39.127136 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.127105 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 12:24:39.179788 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.179762 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6tq6s"] Jul 15 12:24:39.182699 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.182680 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.182812 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.182759 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ml5q4"] Jul 15 12:24:39.185795 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.185777 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.186053 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.186030 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ggrzt\"" Jul 15 12:24:39.186053 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.186056 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 12:24:39.186644 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.186627 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 12:24:39.188643 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.188622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 12:24:39.188735 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.188649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gd95d\"" Jul 15 12:24:39.188906 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.188892 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 12:24:39.189398 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.189380 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 12:24:39.195386 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.195323 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6tq6s"] Jul 15 12:24:39.197424 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.197407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ml5q4"] Jul 15 12:24:39.205031 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.205014 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9vqh8"] Jul 15 12:24:39.213356 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.213340 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.216189 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.216169 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x7vt8\"" Jul 15 12:24:39.216291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.216196 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 12:24:39.216291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.216206 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 12:24:39.216291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.216197 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 12:24:39.216291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.216261 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 12:24:39.220001 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.219983 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9vqh8"] Jul 15 12:24:39.323639 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcfec8d-a62c-4655-9b8e-2f4da02fb933-cert\") pod \"ingress-canary-ml5q4\" (UID: \"9dcfec8d-a62c-4655-9b8e-2f4da02fb933\") " pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f19759-48e5-44e9-94bd-5bdc03461b51-tmp-dir\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323659 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afb2ccd-2367-400d-ac55-e08280a3eaa7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55f19759-48e5-44e9-94bd-5bdc03461b51-config-volume\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f19759-48e5-44e9-94bd-5bdc03461b51-metrics-tls\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvz9g\" (UniqueName: \"kubernetes.io/projected/8afb2ccd-2367-400d-ac55-e08280a3eaa7-kube-api-access-mvz9g\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8afb2ccd-2367-400d-ac55-e08280a3eaa7-crio-socket\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afb2ccd-2367-400d-ac55-e08280a3eaa7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9s6\" (UniqueName: \"kubernetes.io/projected/55f19759-48e5-44e9-94bd-5bdc03461b51-kube-api-access-zv9s6\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.323974 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnshm\" (UniqueName: \"kubernetes.io/projected/9dcfec8d-a62c-4655-9b8e-2f4da02fb933-kube-api-access-hnshm\") pod \"ingress-canary-ml5q4\" (UID: \"9dcfec8d-a62c-4655-9b8e-2f4da02fb933\") " pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.324252 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.323997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afb2ccd-2367-400d-ac55-e08280a3eaa7-data-volume\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.424960 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.424889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcfec8d-a62c-4655-9b8e-2f4da02fb933-cert\") pod \"ingress-canary-ml5q4\" (UID: \"9dcfec8d-a62c-4655-9b8e-2f4da02fb933\") " pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.424960 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.424915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f19759-48e5-44e9-94bd-5bdc03461b51-tmp-dir\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.424960 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.424931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afb2ccd-2367-400d-ac55-e08280a3eaa7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.424960 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.424950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55f19759-48e5-44e9-94bd-5bdc03461b51-config-volume\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.425195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.424974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f19759-48e5-44e9-94bd-5bdc03461b51-metrics-tls\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.425195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.424996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvz9g\" (UniqueName: \"kubernetes.io/projected/8afb2ccd-2367-400d-ac55-e08280a3eaa7-kube-api-access-mvz9g\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.425195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8afb2ccd-2367-400d-ac55-e08280a3eaa7-crio-socket\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.425195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afb2ccd-2367-400d-ac55-e08280a3eaa7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.425350 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9s6\" (UniqueName: \"kubernetes.io/projected/55f19759-48e5-44e9-94bd-5bdc03461b51-kube-api-access-zv9s6\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.425350 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnshm\" (UniqueName: \"kubernetes.io/projected/9dcfec8d-a62c-4655-9b8e-2f4da02fb933-kube-api-access-hnshm\") pod \"ingress-canary-ml5q4\" (UID: \"9dcfec8d-a62c-4655-9b8e-2f4da02fb933\") " pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.425350 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f19759-48e5-44e9-94bd-5bdc03461b51-tmp-dir\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.425350 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afb2ccd-2367-400d-ac55-e08280a3eaa7-data-volume\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.425512 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8afb2ccd-2367-400d-ac55-e08280a3eaa7-crio-socket\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.425627 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afb2ccd-2367-400d-ac55-e08280a3eaa7-data-volume\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.425733 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55f19759-48e5-44e9-94bd-5bdc03461b51-config-volume\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.425733 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.425655 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afb2ccd-2367-400d-ac55-e08280a3eaa7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.428917 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.428899 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f19759-48e5-44e9-94bd-5bdc03461b51-metrics-tls\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.428995 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.428922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afb2ccd-2367-400d-ac55-e08280a3eaa7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.429036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.428993 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcfec8d-a62c-4655-9b8e-2f4da02fb933-cert\") pod \"ingress-canary-ml5q4\" (UID: \"9dcfec8d-a62c-4655-9b8e-2f4da02fb933\") " pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.434468 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.434449 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvz9g\" (UniqueName: \"kubernetes.io/projected/8afb2ccd-2367-400d-ac55-e08280a3eaa7-kube-api-access-mvz9g\") pod \"insights-runtime-extractor-9vqh8\" (UID: \"8afb2ccd-2367-400d-ac55-e08280a3eaa7\") " pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.435320 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.435303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9s6\" (UniqueName: \"kubernetes.io/projected/55f19759-48e5-44e9-94bd-5bdc03461b51-kube-api-access-zv9s6\") pod \"dns-default-6tq6s\" (UID: \"55f19759-48e5-44e9-94bd-5bdc03461b51\") " pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.435757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.435739 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnshm\" (UniqueName: \"kubernetes.io/projected/9dcfec8d-a62c-4655-9b8e-2f4da02fb933-kube-api-access-hnshm\") pod \"ingress-canary-ml5q4\" (UID: \"9dcfec8d-a62c-4655-9b8e-2f4da02fb933\") " pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.493356 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.493335 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:39.499465 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.499451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ml5q4" Jul 15 12:24:39.521994 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.521977 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9vqh8" Jul 15 12:24:39.580393 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.580332 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4e87e0c-8851-4a19-8a10-25e541e5d6f4" containerID="ca2cc6b3c7147e19697ff5e746fa0efa7d18712795ad74800ea672932d8ed2e3" exitCode=0 Jul 15 12:24:39.580393 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.580382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerDied","Data":"ca2cc6b3c7147e19697ff5e746fa0efa7d18712795ad74800ea672932d8ed2e3"} Jul 15 12:24:39.681170 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.681137 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ml5q4"] Jul 15 12:24:39.684656 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.684623 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9vqh8"] Jul 15 12:24:39.685481 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:39.685455 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6tq6s"] Jul 15 12:24:39.686087 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:39.686057 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcfec8d_a62c_4655_9b8e_2f4da02fb933.slice/crio-f6758e44ea876acf071ed57e6277de7d0d3895f5811d16ec2e8039e0927f068e WatchSource:0}: Error finding container f6758e44ea876acf071ed57e6277de7d0d3895f5811d16ec2e8039e0927f068e: Status 404 returned error can't find the container with id f6758e44ea876acf071ed57e6277de7d0d3895f5811d16ec2e8039e0927f068e Jul 15 12:24:39.687876 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:39.687722 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afb2ccd_2367_400d_ac55_e08280a3eaa7.slice/crio-9dca564c9f1760cf01cf0f1b2538f53164b7baab0e2ae8f3f779b93ad0599cd3 WatchSource:0}: Error finding container 9dca564c9f1760cf01cf0f1b2538f53164b7baab0e2ae8f3f779b93ad0599cd3: Status 404 returned error can't find the container with id 9dca564c9f1760cf01cf0f1b2538f53164b7baab0e2ae8f3f779b93ad0599cd3 Jul 15 12:24:39.693515 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:39.693495 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f19759_48e5_44e9_94bd_5bdc03461b51.slice/crio-515a247c6a7384aee26c956328bf6175eceb3eba456774bacb446af93f811a69 WatchSource:0}: Error finding container 515a247c6a7384aee26c956328bf6175eceb3eba456774bacb446af93f811a69: Status 404 returned error can't find the container with id 515a247c6a7384aee26c956328bf6175eceb3eba456774bacb446af93f811a69 Jul 15 12:24:40.308770 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.308583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:24:40.308770 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.308583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:24:40.309011 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.308952 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:24:40.313650 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.313614 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 12:24:40.313757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.313728 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qvxmk\"" Jul 15 12:24:40.313890 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.313870 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 12:24:40.314100 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.313948 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 12:24:40.314100 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.313993 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 12:24:40.314100 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.313948 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2twz\"" Jul 15 12:24:40.589995 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.589918 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" event={"ID":"c4e87e0c-8851-4a19-8a10-25e541e5d6f4","Type":"ContainerStarted","Data":"10f53f84b4db39dafea02df16d922eadba141ab8936dd93fccbe8d53cd67bc03"} Jul 15 12:24:40.592006 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.591978 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9vqh8" event={"ID":"8afb2ccd-2367-400d-ac55-e08280a3eaa7","Type":"ContainerStarted","Data":"9dca564c9f1760cf01cf0f1b2538f53164b7baab0e2ae8f3f779b93ad0599cd3"} Jul 15 12:24:40.593855 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.593785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ml5q4" event={"ID":"9dcfec8d-a62c-4655-9b8e-2f4da02fb933","Type":"ContainerStarted","Data":"f6758e44ea876acf071ed57e6277de7d0d3895f5811d16ec2e8039e0927f068e"} Jul 15 12:24:40.595490 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:40.595466 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6tq6s" event={"ID":"55f19759-48e5-44e9-94bd-5bdc03461b51","Type":"ContainerStarted","Data":"515a247c6a7384aee26c956328bf6175eceb3eba456774bacb446af93f811a69"} Jul 15 12:24:41.740009 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.739728 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rpr5r" podStartSLOduration=10.38233503 podStartE2EDuration="40.739703593s" totalImagesPullingTime="30.357368563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:40.621614375 +0000 UTC m=+39.834360887" watchObservedRunningTime="2026-07-15 12:24:41.739703593 +0000 UTC m=+40.952450104" Jul 15 12:24:41.740009 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.740005 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7"] Jul 15 12:24:41.760338 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.759861 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7"] Jul 15 12:24:41.760338 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.759971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:41.763996 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.763791 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tk5pm\"" Jul 15 12:24:41.763996 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.763908 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 15 12:24:41.944229 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:41.944190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fab19c5-968b-4452-af0d-fcb562c57f43-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-htsr7\" (UID: \"5fab19c5-968b-4452-af0d-fcb562c57f43\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:42.045121 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:42.045091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fab19c5-968b-4452-af0d-fcb562c57f43-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-htsr7\" (UID: \"5fab19c5-968b-4452-af0d-fcb562c57f43\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:42.048929 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:42.048899 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fab19c5-968b-4452-af0d-fcb562c57f43-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-htsr7\" (UID: \"5fab19c5-968b-4452-af0d-fcb562c57f43\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:42.072411 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:42.072385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:45.802417 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.802387 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5d4f5d4d-kzd9g"] Jul 15 12:24:45.848589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.848544 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5d4f5d4d-kzd9g"] Jul 15 12:24:45.848764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.848662 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.855014 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.854911 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 12:24:45.855414 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.855397 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jhjl8\"" Jul 15 12:24:45.856395 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.856381 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 12:24:45.856464 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.856432 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 12:24:45.856543 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.856466 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 12:24:45.856543 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.856518 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 12:24:45.856720 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.856707 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 12:24:45.856826 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.856804 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 12:24:45.862026 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.862008 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 12:24:45.970352 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-service-ca\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.970352 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-trusted-ca-bundle\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.970548 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbr5\" (UniqueName: \"kubernetes.io/projected/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-kube-api-access-spbr5\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.970548 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-serving-cert\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.970548 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-oauth-serving-cert\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.970548 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-config\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:45.970751 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:45.970612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-oauth-config\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-service-ca\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-trusted-ca-bundle\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071663 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbr5\" (UniqueName: \"kubernetes.io/projected/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-kube-api-access-spbr5\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-serving-cert\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071868 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-oauth-serving-cert\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-config\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.071998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-oauth-config\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.072706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.072354 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-service-ca\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.074772 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.073843 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-oauth-serving-cert\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.074772 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.074051 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-config\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.075255 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.075227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-serving-cert\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.075538 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.075518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-trusted-ca-bundle\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.076726 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.076703 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-oauth-config\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.089430 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.089406 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbr5\" (UniqueName: \"kubernetes.io/projected/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-kube-api-access-spbr5\") pod \"console-5d5d4f5d4d-kzd9g\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.157355 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.157309 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:46.410150 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.410102 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7"] Jul 15 12:24:46.426544 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.426518 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5d4f5d4d-kzd9g"] Jul 15 12:24:46.459747 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:46.459654 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fab19c5_968b_4452_af0d_fcb562c57f43.slice/crio-d61e2f7da7bfd9882a39339a8adac5a0f77e2748f93839fb0aff61df0a68198b WatchSource:0}: Error finding container d61e2f7da7bfd9882a39339a8adac5a0f77e2748f93839fb0aff61df0a68198b: Status 404 returned error can't find the container with id d61e2f7da7bfd9882a39339a8adac5a0f77e2748f93839fb0aff61df0a68198b Jul 15 12:24:46.460244 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:46.460218 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee258c14_5ccf_4feb_8cdb_8fe4f92527ec.slice/crio-bff06b5a12be2724f86e47ddd2560342d75e9a6b04de766bef202b8172103b5a WatchSource:0}: Error finding container bff06b5a12be2724f86e47ddd2560342d75e9a6b04de766bef202b8172103b5a: Status 404 returned error can't find the container with id bff06b5a12be2724f86e47ddd2560342d75e9a6b04de766bef202b8172103b5a Jul 15 12:24:46.614171 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.614134 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9vqh8" event={"ID":"8afb2ccd-2367-400d-ac55-e08280a3eaa7","Type":"ContainerStarted","Data":"4678f858287eecc94165ed7e6e2eaf434546416feb0c74a849894dd1242a3899"} Jul 15 12:24:46.615990 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.615963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6tq6s" event={"ID":"55f19759-48e5-44e9-94bd-5bdc03461b51","Type":"ContainerStarted","Data":"a4f2bc7ad365b1312ebedd20d4d53d15ef788d9096492983e95c66f7dc162cbc"} Jul 15 12:24:46.616102 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.616012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6tq6s" event={"ID":"55f19759-48e5-44e9-94bd-5bdc03461b51","Type":"ContainerStarted","Data":"6271c27bd328db561fc797d13d7a82a51b2b878c36f7e3fd4b1e826e51768786"} Jul 15 12:24:46.616102 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.616052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:46.618182 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.618149 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ml5q4" event={"ID":"9dcfec8d-a62c-4655-9b8e-2f4da02fb933","Type":"ContainerStarted","Data":"05c11b72e108622dd7d6ee030d35adcd7b84ce98fb83870d3c4a398956f69289"} Jul 15 12:24:46.619734 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.619702 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d4f5d4d-kzd9g" event={"ID":"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec","Type":"ContainerStarted","Data":"bff06b5a12be2724f86e47ddd2560342d75e9a6b04de766bef202b8172103b5a"} Jul 15 12:24:46.620470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.620451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" event={"ID":"5fab19c5-968b-4452-af0d-fcb562c57f43","Type":"ContainerStarted","Data":"d61e2f7da7bfd9882a39339a8adac5a0f77e2748f93839fb0aff61df0a68198b"} Jul 15 12:24:46.671853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:46.671811 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6tq6s" podStartSLOduration=1.103383461 podStartE2EDuration="7.671798455s" totalImagesPullingTime="6.568414994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:46.641939709 +0000 UTC m=+45.854686221" watchObservedRunningTime="2026-07-15 12:24:46.671798455 +0000 UTC m=+45.884544965" Jul 15 12:24:52.638555 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.638522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9vqh8" event={"ID":"8afb2ccd-2367-400d-ac55-e08280a3eaa7","Type":"ContainerStarted","Data":"f582d236dcf618cf35b742bdc7ea588a0ebfb58442a54a193bbf59359c4b2720"} Jul 15 12:24:52.639903 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.639873 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d4f5d4d-kzd9g" event={"ID":"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec","Type":"ContainerStarted","Data":"36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196"} Jul 15 12:24:52.641262 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.641244 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" event={"ID":"5fab19c5-968b-4452-af0d-fcb562c57f43","Type":"ContainerStarted","Data":"6b5f2d84b6abcc6b7bd7f1bd772023e0b1bf71e1be6a51cf136a8cbe9076470e"} Jul 15 12:24:52.641435 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.641421 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:52.645750 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.645734 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" Jul 15 12:24:52.660674 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.660639 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ml5q4" podStartSLOduration=7.082390995 podStartE2EDuration="13.660627892s" totalImagesPullingTime="6.578236897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:46.672138714 +0000 UTC m=+45.884885223" watchObservedRunningTime="2026-07-15 12:24:52.660627892 +0000 UTC m=+51.873374409" Jul 15 12:24:52.661036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.661007 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5d4f5d4d-kzd9g" podStartSLOduration=1.8329393349999998 podStartE2EDuration="7.660995927s" totalImagesPullingTime="5.828056592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:52.660135295 +0000 UTC m=+51.872882006" watchObservedRunningTime="2026-07-15 12:24:52.660995927 +0000 UTC m=+51.873742498" Jul 15 12:24:52.683237 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:52.683206 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-htsr7" podStartSLOduration=6.606328682 podStartE2EDuration="11.683197405s" totalImagesPullingTime="5.076868723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:52.682925753 +0000 UTC m=+51.895672274" watchObservedRunningTime="2026-07-15 12:24:52.683197405 +0000 UTC m=+51.895943911" Jul 15 12:24:53.842441 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.842409 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-5znvn"] Jul 15 12:24:53.845385 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.845363 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:53.850136 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.850116 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 12:24:53.850221 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.850115 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 12:24:53.851518 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.851493 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 12:24:53.851518 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.851516 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 12:24:53.851736 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.851553 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 12:24:53.851736 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.851579 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-npq7z\"" Jul 15 12:24:53.874154 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.874132 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-5znvn"] Jul 15 12:24:53.933346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.933317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:53.933461 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.933385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:53.933461 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.933432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjh6\" (UniqueName: \"kubernetes.io/projected/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-kube-api-access-nrjh6\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:53.933581 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:53.933520 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.034633 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.034599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.034789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.034642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjh6\" (UniqueName: \"kubernetes.io/projected/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-kube-api-access-nrjh6\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.034789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.034673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.034789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.034735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.035467 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.035437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.037235 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.037198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.037335 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.037250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.046108 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.046084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjh6\" (UniqueName: \"kubernetes.io/projected/cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c-kube-api-access-nrjh6\") pod \"prometheus-operator-64785b6675-5znvn\" (UID: \"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.155880 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.155816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" Jul 15 12:24:54.297526 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.297493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-5znvn"] Jul 15 12:24:54.300686 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:54.300658 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb9385e_2f4c_4b07_9e32_d75ee6ceda4c.slice/crio-7eacf5e2db29fcb840b739f0a42b4b689158272047b350322c2e84498bee26bc WatchSource:0}: Error finding container 7eacf5e2db29fcb840b739f0a42b4b689158272047b350322c2e84498bee26bc: Status 404 returned error can't find the container with id 7eacf5e2db29fcb840b739f0a42b4b689158272047b350322c2e84498bee26bc Jul 15 12:24:54.651397 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:54.651324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" event={"ID":"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c","Type":"ContainerStarted","Data":"7eacf5e2db29fcb840b739f0a42b4b689158272047b350322c2e84498bee26bc"} Jul 15 12:24:56.158212 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.158174 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:56.158605 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.158224 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:24:56.159513 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.159492 2582 patch_prober.go:28] interesting pod/console-5d5d4f5d4d-kzd9g container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 15 12:24:56.159578 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.159533 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d5d4f5d4d-kzd9g" podUID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 15 12:24:56.626547 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.626469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6tq6s" Jul 15 12:24:56.659140 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.659104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9vqh8" event={"ID":"8afb2ccd-2367-400d-ac55-e08280a3eaa7","Type":"ContainerStarted","Data":"5f26b0e2296b0f40fce1749eaa2f780dc3dd21171a91dc64e6f26949fc0fe9a9"} Jul 15 12:24:56.684653 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:56.684595 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9vqh8" podStartSLOduration=1.335239609 podStartE2EDuration="17.68455458s" totalImagesPullingTime="16.349314971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:56.684060071 +0000 UTC m=+55.896806581" watchObservedRunningTime="2026-07-15 12:24:56.68455458 +0000 UTC m=+55.897301090" Jul 15 12:24:57.664141 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:57.664105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" event={"ID":"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c","Type":"ContainerStarted","Data":"92b4e176a27269ed8b76432de930656817c53d08bf6802292346502eaaaca63e"} Jul 15 12:24:57.664598 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:57.664168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" event={"ID":"cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c","Type":"ContainerStarted","Data":"1b14b4bb9c0d3797ef9e42823392d72713e91cf30f4a044910441bc1b0583a04"} Jul 15 12:24:57.686039 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:57.685994 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-5znvn" podStartSLOduration=1.9346722330000001 podStartE2EDuration="4.685980213s" totalImagesPullingTime="2.75130798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:57.685174248 +0000 UTC m=+56.897920757" watchObservedRunningTime="2026-07-15 12:24:57.685980213 +0000 UTC m=+56.898726722" Jul 15 12:24:59.508805 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.508776 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7"] Jul 15 12:24:59.542904 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.542872 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7"] Jul 15 12:24:59.543046 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.543001 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.547161 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.547138 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d524z"] Jul 15 12:24:59.548850 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.548825 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-96m58\"" Jul 15 12:24:59.549009 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.548983 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 12:24:59.554582 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.554544 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 12:24:59.565678 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.565660 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dc75v"] Jul 15 12:24:59.565796 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.565782 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.568638 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.568619 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 12:24:59.568749 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.568734 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 12:24:59.569145 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.569130 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 12:24:59.569145 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.569139 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2f47l\"" Jul 15 12:24:59.574852 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.574833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5sdf\" (UniqueName: \"kubernetes.io/projected/772fd62e-7c74-46d2-8907-0e822a22b6f1-kube-api-access-j5sdf\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.574976 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.574862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.574976 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.574885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/772fd62e-7c74-46d2-8907-0e822a22b6f1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.574976 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.574911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.579782 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.579760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.581432 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.581412 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dc75v"] Jul 15 12:24:59.582855 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.582834 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 12:24:59.582949 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.582857 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 12:24:59.582949 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.582885 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 12:24:59.582949 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.582887 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t9v6f\"" Jul 15 12:24:59.675367 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675341 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675475 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-accelerators-collector-config\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675475 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675400 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-metrics-client-ca\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675475 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.675589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5sdf\" (UniqueName: \"kubernetes.io/projected/772fd62e-7c74-46d2-8907-0e822a22b6f1-kube-api-access-j5sdf\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.675589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d43a4a-fded-403e-8ba1-c6c2943e4dee-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.675589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.675589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-textfile\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-root\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.675757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/772fd62e-7c74-46d2-8907-0e822a22b6f1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.675757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31d43a4a-fded-403e-8ba1-c6c2943e4dee-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.675757 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclr2\" (UniqueName: \"kubernetes.io/projected/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-api-access-qclr2\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.675757 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:59.675655 2582 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 15 12:24:59.675913 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp8f\" (UniqueName: \"kubernetes.io/projected/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-kube-api-access-6hp8f\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675913 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:24:59.675797 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-tls podName:772fd62e-7c74-46d2-8907-0e822a22b6f1 nodeName:}" failed. No retries permitted until 2026-07-15 12:25:00.175781603 +0000 UTC m=+59.388528092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-tkkn7" (UID: "772fd62e-7c74-46d2-8907-0e822a22b6f1") : secret "openshift-state-metrics-tls" not found Jul 15 12:24:59.675913 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675871 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-sys\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.675913 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-tls\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.676029 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.676029 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.675936 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-wtmp\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.676093 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.676064 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.676211 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.676186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/772fd62e-7c74-46d2-8907-0e822a22b6f1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.678002 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.677980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.689805 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.689787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5sdf\" (UniqueName: \"kubernetes.io/projected/772fd62e-7c74-46d2-8907-0e822a22b6f1-kube-api-access-j5sdf\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:24:59.776815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.776764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp8f\" (UniqueName: \"kubernetes.io/projected/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-kube-api-access-6hp8f\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.776815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.776792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-sys\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.776815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.776808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-tls\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.776954 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.776823 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-wtmp\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.776954 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.776919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-wtmp\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777015 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.776948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-sys\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777046 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.777094 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-accelerators-collector-config\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777162 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-metrics-client-ca\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777213 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.777265 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d43a4a-fded-403e-8ba1-c6c2943e4dee-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.777265 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-textfile\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-root\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.777346 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31d43a4a-fded-403e-8ba1-c6c2943e4dee-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.777484 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclr2\" (UniqueName: \"kubernetes.io/projected/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-api-access-qclr2\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.777484 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777462 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-root\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777765 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777742 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-accelerators-collector-config\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.777765 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777746 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.778013 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.777990 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-textfile\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.778078 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.778023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31d43a4a-fded-403e-8ba1-c6c2943e4dee-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.778078 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.778062 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d43a4a-fded-403e-8ba1-c6c2943e4dee-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.778237 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.778218 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-metrics-client-ca\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.779325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.779308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-tls\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.779557 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.779535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.779967 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.779949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.780035 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.779973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.789764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.789747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclr2\" (UniqueName: \"kubernetes.io/projected/31d43a4a-fded-403e-8ba1-c6c2943e4dee-kube-api-access-qclr2\") pod \"kube-state-metrics-785778ff74-dc75v\" (UID: \"31d43a4a-fded-403e-8ba1-c6c2943e4dee\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.791509 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.791490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp8f\" (UniqueName: \"kubernetes.io/projected/99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1-kube-api-access-6hp8f\") pod \"node-exporter-d524z\" (UID: \"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1\") " pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.872988 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.872967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d524z" Jul 15 12:24:59.888247 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:24:59.888232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" Jul 15 12:24:59.908495 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:24:59.908472 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a7d5ee_0d6a_4b8f_92af_762f25ce7bf1.slice/crio-b4cbf5eb9de08b9695bd5f23b78839ef7959cd5d1d3ba87438148ee9d7ca217f WatchSource:0}: Error finding container b4cbf5eb9de08b9695bd5f23b78839ef7959cd5d1d3ba87438148ee9d7ca217f: Status 404 returned error can't find the container with id b4cbf5eb9de08b9695bd5f23b78839ef7959cd5d1d3ba87438148ee9d7ca217f Jul 15 12:25:00.060606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.060512 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dc75v"] Jul 15 12:25:00.064136 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:00.064104 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d43a4a_fded_403e_8ba1_c6c2943e4dee.slice/crio-b99eb6478e3db0cf7e656bf9e9d7fbd948ed1efc0121aaa33264da44c43a2c7a WatchSource:0}: Error finding container b99eb6478e3db0cf7e656bf9e9d7fbd948ed1efc0121aaa33264da44c43a2c7a: Status 404 returned error can't find the container with id b99eb6478e3db0cf7e656bf9e9d7fbd948ed1efc0121aaa33264da44c43a2c7a Jul 15 12:25:00.180955 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.180918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:25:00.183155 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.183138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/772fd62e-7c74-46d2-8907-0e822a22b6f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tkkn7\" (UID: \"772fd62e-7c74-46d2-8907-0e822a22b6f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:25:00.453127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.453047 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" Jul 15 12:25:00.589329 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.589298 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:25:00.620737 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.620708 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.630257 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.630238 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 12:25:00.632453 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.632431 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 12:25:00.632515 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.632473 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 12:25:00.632968 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.632954 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 12:25:00.633431 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.633419 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-79htw\"" Jul 15 12:25:00.634211 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.634195 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 12:25:00.634968 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.634955 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 12:25:00.635027 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.634959 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 12:25:00.635027 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.634988 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 12:25:00.655238 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.655217 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 12:25:00.676406 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.676380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" event={"ID":"31d43a4a-fded-403e-8ba1-c6c2943e4dee","Type":"ContainerStarted","Data":"b99eb6478e3db0cf7e656bf9e9d7fbd948ed1efc0121aaa33264da44c43a2c7a"} Jul 15 12:25:00.677705 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.677654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d524z" event={"ID":"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1","Type":"ContainerStarted","Data":"b4cbf5eb9de08b9695bd5f23b78839ef7959cd5d1d3ba87438148ee9d7ca217f"} Jul 15 12:25:00.685093 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7873b582-119c-4b6a-b68e-4238029dd533-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685093 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685075 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-config-volume\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685168 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7873b582-119c-4b6a-b68e-4238029dd533-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685200 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqbq\" (UniqueName: \"kubernetes.io/projected/7873b582-119c-4b6a-b68e-4238029dd533-kube-api-access-8bqbq\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685440 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7873b582-119c-4b6a-b68e-4238029dd533-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685440 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7873b582-119c-4b6a-b68e-4238029dd533-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685440 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7873b582-119c-4b6a-b68e-4238029dd533-config-out\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685440 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-web-config\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685594 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.685594 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.685491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.695838 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:00.695801 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772fd62e_7c74_46d2_8907_0e822a22b6f1.slice/crio-83d7cb19ab11698bf4931fb3172dc6e840f31cdf20015560520612d772172715 WatchSource:0}: Error finding container 83d7cb19ab11698bf4931fb3172dc6e840f31cdf20015560520612d772172715: Status 404 returned error can't find the container with id 83d7cb19ab11698bf4931fb3172dc6e840f31cdf20015560520612d772172715 Jul 15 12:25:00.697888 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.697780 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7"] Jul 15 12:25:00.755593 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.753167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:25:00.788406 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788406 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-config-volume\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788573 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788573 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7873b582-119c-4b6a-b68e-4238029dd533-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788573 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqbq\" (UniqueName: \"kubernetes.io/projected/7873b582-119c-4b6a-b68e-4238029dd533-kube-api-access-8bqbq\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788573 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788702 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7873b582-119c-4b6a-b68e-4238029dd533-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788702 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7873b582-119c-4b6a-b68e-4238029dd533-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788788 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7873b582-119c-4b6a-b68e-4238029dd533-config-out\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788788 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-web-config\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.788788 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.788764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.789944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.789240 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.789944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.789298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7873b582-119c-4b6a-b68e-4238029dd533-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.789944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.789655 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7873b582-119c-4b6a-b68e-4238029dd533-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.789944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.789755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7873b582-119c-4b6a-b68e-4238029dd533-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.790285 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.790032 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7873b582-119c-4b6a-b68e-4238029dd533-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.791614 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.791543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7873b582-119c-4b6a-b68e-4238029dd533-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.794331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.794121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.794331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.794170 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-config-volume\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.794478 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.794408 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.794538 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.794508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.794692 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.794639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.795331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.795313 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.796161 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.796095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7873b582-119c-4b6a-b68e-4238029dd533-web-config\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.796161 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.796117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7873b582-119c-4b6a-b68e-4238029dd533-config-out\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.804456 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.804432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqbq\" (UniqueName: \"kubernetes.io/projected/7873b582-119c-4b6a-b68e-4238029dd533-kube-api-access-8bqbq\") pod \"alertmanager-main-0\" (UID: \"7873b582-119c-4b6a-b68e-4238029dd533\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:00.929654 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:00.929606 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:25:01.179069 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.179029 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:25:01.402846 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:01.402759 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7873b582_119c_4b6a_b68e_4238029dd533.slice/crio-df074976e18a835418db5e567a2282c9e1ccd2afb9d91996ef6c4a8bcccaa6af WatchSource:0}: Error finding container df074976e18a835418db5e567a2282c9e1ccd2afb9d91996ef6c4a8bcccaa6af: Status 404 returned error can't find the container with id df074976e18a835418db5e567a2282c9e1ccd2afb9d91996ef6c4a8bcccaa6af Jul 15 12:25:01.684011 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.683904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"df074976e18a835418db5e567a2282c9e1ccd2afb9d91996ef6c4a8bcccaa6af"} Jul 15 12:25:01.686038 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.685987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" event={"ID":"772fd62e-7c74-46d2-8907-0e822a22b6f1","Type":"ContainerStarted","Data":"ba5e83019b48cafeba5a2b2ef7c2c06e39ee31dc573476ec1ad9a5c0e298e84c"} Jul 15 12:25:01.686169 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.686045 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" event={"ID":"772fd62e-7c74-46d2-8907-0e822a22b6f1","Type":"ContainerStarted","Data":"a4f27ef54ff78191796b9a4f46a630d78dd0adc1a9310a43cb13f2893d8a0f64"} Jul 15 12:25:01.686169 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.686077 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" event={"ID":"772fd62e-7c74-46d2-8907-0e822a22b6f1","Type":"ContainerStarted","Data":"83d7cb19ab11698bf4931fb3172dc6e840f31cdf20015560520612d772172715"} Jul 15 12:25:01.688851 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.688740 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" event={"ID":"31d43a4a-fded-403e-8ba1-c6c2943e4dee","Type":"ContainerStarted","Data":"5ee33e27e58d45ac172fd577e27bdbe11aba7395044e77d44f01f395f6a3868b"} Jul 15 12:25:01.688851 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.688791 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" event={"ID":"31d43a4a-fded-403e-8ba1-c6c2943e4dee","Type":"ContainerStarted","Data":"67fcadd29b6c2fe729b4c32c356ebfc57db8719ab8a27895cc319b47b1c5a991"} Jul 15 12:25:01.688851 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.688819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" event={"ID":"31d43a4a-fded-403e-8ba1-c6c2943e4dee","Type":"ContainerStarted","Data":"5b02a8ca4a83abea5ce8953e2d03d2ea6a9e05ab3a300609d1a9c47f1e70b770"} Jul 15 12:25:01.690939 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.690831 2582 generic.go:356] "Generic (PLEG): container finished" podID="99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1" containerID="31291aeb4cfa8bde3e93ce792ac3319a7129af4ef959f3d4d9edbdc4c9b23b88" exitCode=0 Jul 15 12:25:01.690939 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.690893 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d524z" event={"ID":"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1","Type":"ContainerDied","Data":"31291aeb4cfa8bde3e93ce792ac3319a7129af4ef959f3d4d9edbdc4c9b23b88"} Jul 15 12:25:01.802635 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:01.802369 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-dc75v" podStartSLOduration=1.425540499 podStartE2EDuration="2.802348618s" totalImagesPullingTime="1.376808119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:01.758057989 +0000 UTC m=+60.970804503" watchObservedRunningTime="2026-07-15 12:25:01.802348618 +0000 UTC m=+61.015095129" Jul 15 12:25:02.552573 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.552534 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6qvz" Jul 15 12:25:02.564717 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.564688 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l"] Jul 15 12:25:02.568340 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.568321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.571575 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.571539 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 12:25:02.571784 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.571768 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 12:25:02.571857 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.571840 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fm5xr\"" Jul 15 12:25:02.571897 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.571858 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cb5g91suhefae\"" Jul 15 12:25:02.571964 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.571953 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 12:25:02.572056 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.572043 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 12:25:02.572104 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.572092 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 12:25:02.593275 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.593249 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l"] Jul 15 12:25:02.605685 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605664 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.605820 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.605820 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc63c38d-1ab0-4386-89a2-11cda137d06c-metrics-client-ca\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.605932 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-grpc-tls\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.605932 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.606034 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605958 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.606034 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.605995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-tls\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.606034 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.606025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxx88\" (UniqueName: \"kubernetes.io/projected/dc63c38d-1ab0-4386-89a2-11cda137d06c-kube-api-access-pxx88\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.696760 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.696676 2582 generic.go:356] "Generic (PLEG): container finished" podID="7873b582-119c-4b6a-b68e-4238029dd533" containerID="36a5e9ad8f6aebc0ebb5ab6e20157e4c131fd74eeecd740090768fc177aa33b5" exitCode=0 Jul 15 12:25:02.696760 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.696753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerDied","Data":"36a5e9ad8f6aebc0ebb5ab6e20157e4c131fd74eeecd740090768fc177aa33b5"} Jul 15 12:25:02.699002 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.698979 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" event={"ID":"772fd62e-7c74-46d2-8907-0e822a22b6f1","Type":"ContainerStarted","Data":"1d13ef6b6092946b0f7581c4a76c4c43131881e6dde9131872bb2232cfea9272"} Jul 15 12:25:02.701034 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.701011 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d524z" event={"ID":"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1","Type":"ContainerStarted","Data":"551e57e090926a3ec73d59d10545dbf92fa978c7f2f32ee4edd85349b92b9b9d"} Jul 15 12:25:02.701116 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.701057 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d524z" event={"ID":"99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1","Type":"ContainerStarted","Data":"0892b5fa5a4e7aa1cb6a146fca0dfb815680f43858daa544982ca3655b76d7ac"} Jul 15 12:25:02.707137 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707218 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707218 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc63c38d-1ab0-4386-89a2-11cda137d06c-metrics-client-ca\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707218 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-grpc-tls\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707358 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707358 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707358 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-tls\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707358 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxx88\" (UniqueName: \"kubernetes.io/projected/dc63c38d-1ab0-4386-89a2-11cda137d06c-kube-api-access-pxx88\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.707983 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.707956 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc63c38d-1ab0-4386-89a2-11cda137d06c-metrics-client-ca\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.709873 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.709826 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-grpc-tls\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.710176 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.710069 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.710176 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.710070 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-tls\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.710176 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.710162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.710499 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.710469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.710756 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.710731 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dc63c38d-1ab0-4386-89a2-11cda137d06c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.722366 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.722348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxx88\" (UniqueName: \"kubernetes.io/projected/dc63c38d-1ab0-4386-89a2-11cda137d06c-kube-api-access-pxx88\") pod \"thanos-querier-5b6b6f9d9c-59h2l\" (UID: \"dc63c38d-1ab0-4386-89a2-11cda137d06c\") " pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:02.779292 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.779244 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tkkn7" podStartSLOduration=2.455046623 podStartE2EDuration="3.779230594s" totalImagesPullingTime="1.324183971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:02.773455421 +0000 UTC m=+61.986201931" watchObservedRunningTime="2026-07-15 12:25:02.779230594 +0000 UTC m=+61.991977152" Jul 15 12:25:02.813321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.813270 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d524z" podStartSLOduration=2.605943897 podStartE2EDuration="3.813256834s" totalImagesPullingTime="1.129058824s" totalInitContainerRuntime="78.254113ms" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:02.812746281 +0000 UTC m=+62.025492814" watchObservedRunningTime="2026-07-15 12:25:02.813256834 +0000 UTC m=+62.026003344" Jul 15 12:25:02.880041 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:02.880007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:03.024378 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.024310 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l"] Jul 15 12:25:03.026814 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:03.026784 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc63c38d_1ab0_4386_89a2_11cda137d06c.slice/crio-664c7335daa2d3069d160b177cf1774815ce8f4c21b92580c36d27febaf3fcd2 WatchSource:0}: Error finding container 664c7335daa2d3069d160b177cf1774815ce8f4c21b92580c36d27febaf3fcd2: Status 404 returned error can't find the container with id 664c7335daa2d3069d160b177cf1774815ce8f4c21b92580c36d27febaf3fcd2 Jul 15 12:25:03.707861 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.707772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"664c7335daa2d3069d160b177cf1774815ce8f4c21b92580c36d27febaf3fcd2"} Jul 15 12:25:03.894321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.894289 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bc55c645c-qt8fh"] Jul 15 12:25:03.897664 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.897643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:03.900919 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.900891 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 12:25:03.901041 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.900891 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 12:25:03.901264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.901242 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7g5gg\"" Jul 15 12:25:03.902230 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.902209 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 12:25:03.902648 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.902479 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-kqgtj2uuesd9\"" Jul 15 12:25:03.902751 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.902733 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 12:25:03.910325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:03.910285 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bc55c645c-qt8fh"] Jul 15 12:25:04.019416 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6790754-343b-45c7-bbda-f664360f8d96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.019608 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-secret-metrics-server-tls\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.019608 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlfz\" (UniqueName: \"kubernetes.io/projected/a6790754-343b-45c7-bbda-f664360f8d96-kube-api-access-qnlfz\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.019608 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a6790754-343b-45c7-bbda-f664360f8d96-audit-log\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.019608 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-secret-metrics-server-client-certs\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.019835 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-client-ca-bundle\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.019835 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.019682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a6790754-343b-45c7-bbda-f664360f8d96-metrics-server-audit-profiles\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.083786 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.083751 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5d4f5d4d-kzd9g"] Jul 15 12:25:04.088966 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.088910 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf"] Jul 15 12:25:04.093473 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.092953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.097220 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.096838 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cqq52\"" Jul 15 12:25:04.097220 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.097055 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 12:25:04.107988 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.107966 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf"] Jul 15 12:25:04.120937 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.120903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6790754-343b-45c7-bbda-f664360f8d96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121056 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.120947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-secret-metrics-server-tls\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121056 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.120987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlfz\" (UniqueName: \"kubernetes.io/projected/a6790754-343b-45c7-bbda-f664360f8d96-kube-api-access-qnlfz\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121056 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.121027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a6790754-343b-45c7-bbda-f664360f8d96-audit-log\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121056 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.121051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-secret-metrics-server-client-certs\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121255 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.121076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-client-ca-bundle\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121255 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.121147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a6790754-343b-45c7-bbda-f664360f8d96-metrics-server-audit-profiles\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121712 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.121656 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a6790754-343b-45c7-bbda-f664360f8d96-audit-log\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.121831 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.121789 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6790754-343b-45c7-bbda-f664360f8d96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.122105 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.122085 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a6790754-343b-45c7-bbda-f664360f8d96-metrics-server-audit-profiles\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.123680 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.123652 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-secret-metrics-server-tls\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.123928 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.123858 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-secret-metrics-server-client-certs\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.124549 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.124529 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a6790754-343b-45c7-bbda-f664360f8d96-client-ca-bundle\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.138847 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.138827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlfz\" (UniqueName: \"kubernetes.io/projected/a6790754-343b-45c7-bbda-f664360f8d96-kube-api-access-qnlfz\") pod \"metrics-server-7bc55c645c-qt8fh\" (UID: \"a6790754-343b-45c7-bbda-f664360f8d96\") " pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.153478 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.153450 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f9bbff84c-27p2j"] Jul 15 12:25:04.158064 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.158041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.170279 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.170255 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9bbff84c-27p2j"] Jul 15 12:25:04.210176 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.210150 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:04.222390 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-oauth-config\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.222491 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fa1abfd-d881-4cfe-80ce-b6892e879c84-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-mkhdf\" (UID: \"3fa1abfd-d881-4cfe-80ce-b6892e879c84\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.222491 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqvs\" (UniqueName: \"kubernetes.io/projected/3fa1abfd-d881-4cfe-80ce-b6892e879c84-kube-api-access-5tqvs\") pod \"monitoring-plugin-5f68df7f58-mkhdf\" (UID: \"3fa1abfd-d881-4cfe-80ce-b6892e879c84\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.222491 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncjj\" (UniqueName: \"kubernetes.io/projected/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-kube-api-access-gncjj\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.222663 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-serving-cert\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.222663 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-config\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.222663 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-trusted-ca-bundle\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.222766 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-service-ca\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.222766 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.222717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-oauth-serving-cert\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323340 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-trusted-ca-bundle\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323340 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-service-ca\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323340 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-oauth-serving-cert\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323353 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-oauth-config\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fa1abfd-d881-4cfe-80ce-b6892e879c84-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-mkhdf\" (UID: \"3fa1abfd-d881-4cfe-80ce-b6892e879c84\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.323658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323425 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqvs\" (UniqueName: \"kubernetes.io/projected/3fa1abfd-d881-4cfe-80ce-b6892e879c84-kube-api-access-5tqvs\") pod \"monitoring-plugin-5f68df7f58-mkhdf\" (UID: \"3fa1abfd-d881-4cfe-80ce-b6892e879c84\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.323658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncjj\" (UniqueName: \"kubernetes.io/projected/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-kube-api-access-gncjj\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-serving-cert\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.323658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.323646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-config\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.324132 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.324100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-oauth-serving-cert\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.324683 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.324664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-trusted-ca-bundle\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.324809 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.324788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-config\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.324905 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.324861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-service-ca\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.326200 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.326176 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fa1abfd-d881-4cfe-80ce-b6892e879c84-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-mkhdf\" (UID: \"3fa1abfd-d881-4cfe-80ce-b6892e879c84\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.326276 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.326262 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-oauth-config\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.326710 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.326684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-serving-cert\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.333087 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.333045 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncjj\" (UniqueName: \"kubernetes.io/projected/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-kube-api-access-gncjj\") pod \"console-f9bbff84c-27p2j\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:04.333796 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.333777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqvs\" (UniqueName: \"kubernetes.io/projected/3fa1abfd-d881-4cfe-80ce-b6892e879c84-kube-api-access-5tqvs\") pod \"monitoring-plugin-5f68df7f58-mkhdf\" (UID: \"3fa1abfd-d881-4cfe-80ce-b6892e879c84\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.405803 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.405760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:04.468545 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:04.468507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:05.108448 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.108419 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf"] Jul 15 12:25:05.110615 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:05.110585 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa1abfd_d881_4cfe_80ce_b6892e879c84.slice/crio-dc9731c35d62207bf6aa682327d5f8a6b2890764f08fe5463a8e37192d839606 WatchSource:0}: Error finding container dc9731c35d62207bf6aa682327d5f8a6b2890764f08fe5463a8e37192d839606: Status 404 returned error can't find the container with id dc9731c35d62207bf6aa682327d5f8a6b2890764f08fe5463a8e37192d839606 Jul 15 12:25:05.126607 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.126514 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9bbff84c-27p2j"] Jul 15 12:25:05.130699 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:05.130673 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda51b7c0_dca9_46d1_a66b_f3c8c5a338bb.slice/crio-9f5353d815a33d48aef68469a597adce10295bdc8cb917ef0395fbfdb7f9628f WatchSource:0}: Error finding container 9f5353d815a33d48aef68469a597adce10295bdc8cb917ef0395fbfdb7f9628f: Status 404 returned error can't find the container with id 9f5353d815a33d48aef68469a597adce10295bdc8cb917ef0395fbfdb7f9628f Jul 15 12:25:05.144263 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.144236 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bc55c645c-qt8fh"] Jul 15 12:25:05.149665 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:05.149635 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6790754_343b_45c7_bbda_f664360f8d96.slice/crio-9534750308aa00901b27e011017effb4afb33504886e64e1fb91c8b223c3034f WatchSource:0}: Error finding container 9534750308aa00901b27e011017effb4afb33504886e64e1fb91c8b223c3034f: Status 404 returned error can't find the container with id 9534750308aa00901b27e011017effb4afb33504886e64e1fb91c8b223c3034f Jul 15 12:25:05.649815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.649692 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:25:05.657992 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.657968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.670061 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.667914 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 12:25:05.670061 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.667949 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 12:25:05.671302 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.671281 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 12:25:05.671793 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.671770 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 12:25:05.671997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.671980 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 12:25:05.672306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.672289 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 12:25:05.672379 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.672364 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 12:25:05.672490 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.672473 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 12:25:05.672541 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.672525 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 12:25:05.672762 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.672289 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 12:25:05.672852 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.672837 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 12:25:05.673041 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.673024 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-89pqm\"" Jul 15 12:25:05.676867 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.673523 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 12:25:05.678495 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.677790 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7n94beqm50b32\"" Jul 15 12:25:05.681635 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.680051 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 12:25:05.682606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.682483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:25:05.683237 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.683179 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 12:25:05.734879 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.734633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"bbb68e09bf3037b1237b91771fe75084ea973b2acf7d603b1763dc436eaaeac3"} Jul 15 12:25:05.734879 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.734724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"9e69010c3bd7c1a92f3bc0da4a89818272a5a1704cfb2081e2a8b23c078968b2"} Jul 15 12:25:05.734879 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.734762 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"9322080d649925fd125823c3531c8e5d0b5ebc74c2b04c779c1f5aef2d38103f"} Jul 15 12:25:05.734879 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.734802 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"99bf05786d15cc762d3c6adc532b86df093161ada3a7b4776a0081d445395efc"} Jul 15 12:25:05.734879 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.734836 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"36ecd06c48c9de8fbf0b8de2e661cfe92cd4de6e70b0d67d60df68eafb549f53"} Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737334 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8s5\" (UniqueName: \"kubernetes.io/projected/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-kube-api-access-jq8s5\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737363 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-config\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737505 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738165 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.738853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.737926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.739182 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.738942 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"10b33466be46fc6249544ae95fe1dda21718434531616e0e98092fdce7384e9d"} Jul 15 12:25:05.739182 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.739010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"c6ab8781c03dea7ab3e5bb774dc710a9f83dd1559ae8553a77f266b0a9153181"} Jul 15 12:25:05.739182 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.739050 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"a57b3fbab74d6dd3d4a2b0ac86f084d5d1ff92a0ed3cfa71dbf5e954c05e3676"} Jul 15 12:25:05.744818 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.744051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9bbff84c-27p2j" event={"ID":"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb","Type":"ContainerStarted","Data":"2a9611b3d3fac43dd39aca151d66ce7b36440520a6c638b4622baab53e57180e"} Jul 15 12:25:05.744818 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.744115 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9bbff84c-27p2j" event={"ID":"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb","Type":"ContainerStarted","Data":"9f5353d815a33d48aef68469a597adce10295bdc8cb917ef0395fbfdb7f9628f"} Jul 15 12:25:05.753188 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.753159 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" event={"ID":"a6790754-343b-45c7-bbda-f664360f8d96","Type":"ContainerStarted","Data":"9534750308aa00901b27e011017effb4afb33504886e64e1fb91c8b223c3034f"} Jul 15 12:25:05.761523 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.761491 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" event={"ID":"3fa1abfd-d881-4cfe-80ce-b6892e879c84","Type":"ContainerStarted","Data":"dc9731c35d62207bf6aa682327d5f8a6b2890764f08fe5463a8e37192d839606"} Jul 15 12:25:05.772893 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.772845 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f9bbff84c-27p2j" podStartSLOduration=1.772831917 podStartE2EDuration="1.772831917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:05.772533361 +0000 UTC m=+64.985279874" watchObservedRunningTime="2026-07-15 12:25:05.772831917 +0000 UTC m=+64.985578417" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.838976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.839368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8s5\" (UniqueName: \"kubernetes.io/projected/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-kube-api-access-jq8s5\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840214 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-config\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840214 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840214 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840214 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.839319 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840457 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.840244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840663 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.840592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840663 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.840635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.840815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.840765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.841168 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.841142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.842317 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.841416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.842317 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.841872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.846199 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.842981 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-config\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.847950 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.846320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.847950 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.846921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.847950 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.847283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.847950 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.847774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.847950 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.847911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.848723 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.848471 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.849686 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.849635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.849784 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.849770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.850364 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.850274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.851703 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.850990 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.851703 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.851609 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.851858 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.851817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.851977 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.851912 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:05.853393 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:05.853268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8s5\" (UniqueName: \"kubernetes.io/projected/8f31fc10-6a7f-4d8d-bcc8-05a08b776761-kube-api-access-jq8s5\") pod \"prometheus-k8s-0\" (UID: \"8f31fc10-6a7f-4d8d-bcc8-05a08b776761\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:06.003260 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.003173 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:06.179929 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.179835 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:25:06.768432 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.768402 2582 generic.go:356] "Generic (PLEG): container finished" podID="8f31fc10-6a7f-4d8d-bcc8-05a08b776761" containerID="1e9169ce9f6f764f336a52bf0b3e64aec4365a7e9f14d451933d97e932bac504" exitCode=0 Jul 15 12:25:06.768605 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.768440 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerDied","Data":"1e9169ce9f6f764f336a52bf0b3e64aec4365a7e9f14d451933d97e932bac504"} Jul 15 12:25:06.768605 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.768519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"c393159a4cbab9e8e80e4f3a89f98eb4d19e1192bcaf6ecd80668c5221d1cfcf"} Jul 15 12:25:06.773378 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.773304 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7873b582-119c-4b6a-b68e-4238029dd533","Type":"ContainerStarted","Data":"ce3586b0abf49c6e64da5172114ae7a26261b5428919de9486a57647f2252d97"} Jul 15 12:25:06.776467 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.776415 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"386b5567de76d8cea6d3fd865de247dc8dc28cd7f55883a03a08415a635cd59f"} Jul 15 12:25:06.776576 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.776530 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"c6ad40b17d3e6510516689fccf7a0fea04f201adf179c2fbf562200c5c991d64"} Jul 15 12:25:06.856841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.856745 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.802111307 podStartE2EDuration="6.856723165s" totalImagesPullingTime="4.034640462s" totalInitContainerRuntime="19.971396ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:06.855467529 +0000 UTC m=+66.068214038" watchObservedRunningTime="2026-07-15 12:25:06.856723165 +0000 UTC m=+66.069469677" Jul 15 12:25:06.952869 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.952842 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:25:06.956663 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.956642 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 12:25:06.966075 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:06.966051 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d23692c8-7189-494b-aa94-15feeeda6119-original-pull-secret\") pod \"global-pull-secret-syncer-x9hqp\" (UID: \"d23692c8-7189-494b-aa94-15feeeda6119\") " pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:25:07.032283 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.032263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9hqp" Jul 15 12:25:07.053771 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.053230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:25:07.057222 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.057202 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 12:25:07.066397 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.066376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bac588-12e5-410d-8ea4-cb4ec9bbca1a-metrics-certs\") pod \"network-metrics-daemon-gvnfn\" (UID: \"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a\") " pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:25:07.153913 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.153879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:25:07.159657 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.159632 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 12:25:07.168108 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.168089 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 12:25:07.177166 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.177149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtjx\" (UniqueName: \"kubernetes.io/projected/7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc-kube-api-access-rgtjx\") pod \"network-check-target-tvnkk\" (UID: \"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc\") " pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:25:07.194820 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.194790 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9hqp"] Jul 15 12:25:07.199135 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:07.199114 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23692c8_7189_494b_aa94_15feeeda6119.slice/crio-8a22163ec9af254142a2ff96027484a4246543e6991a92d014679a4c6cfc3ca3 WatchSource:0}: Error finding container 8a22163ec9af254142a2ff96027484a4246543e6991a92d014679a4c6cfc3ca3: Status 404 returned error can't find the container with id 8a22163ec9af254142a2ff96027484a4246543e6991a92d014679a4c6cfc3ca3 Jul 15 12:25:07.327216 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.327177 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qvxmk\"" Jul 15 12:25:07.334883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.334861 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:25:07.343280 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.343255 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2twz\"" Jul 15 12:25:07.348984 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.348942 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvnfn" Jul 15 12:25:07.500285 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.500261 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvnkk"] Jul 15 12:25:07.501820 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:07.501786 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1b3b6a_52c4_425f_8ecb_a1af3d5a2dbc.slice/crio-0f2ee5676377e40869d0d19d2937d3826ca6e2797abed55f2870b13f044da0a3 WatchSource:0}: Error finding container 0f2ee5676377e40869d0d19d2937d3826ca6e2797abed55f2870b13f044da0a3: Status 404 returned error can't find the container with id 0f2ee5676377e40869d0d19d2937d3826ca6e2797abed55f2870b13f044da0a3 Jul 15 12:25:07.524325 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.524304 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvnfn"] Jul 15 12:25:07.525240 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:25:07.525211 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bac588_12e5_410d_8ea4_cb4ec9bbca1a.slice/crio-394ee4c16c80a36475aaf511d0e04557435a280fdfee7b80da42db9171f1ccb8 WatchSource:0}: Error finding container 394ee4c16c80a36475aaf511d0e04557435a280fdfee7b80da42db9171f1ccb8: Status 404 returned error can't find the container with id 394ee4c16c80a36475aaf511d0e04557435a280fdfee7b80da42db9171f1ccb8 Jul 15 12:25:07.782371 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.782331 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvnfn" event={"ID":"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a","Type":"ContainerStarted","Data":"394ee4c16c80a36475aaf511d0e04557435a280fdfee7b80da42db9171f1ccb8"} Jul 15 12:25:07.783931 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.783900 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" event={"ID":"a6790754-343b-45c7-bbda-f664360f8d96","Type":"ContainerStarted","Data":"9d893a9e9364015f3125cc092a61f3c87f2a777daab7e05971f0372b782ae080"} Jul 15 12:25:07.785088 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.785068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvnkk" event={"ID":"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc","Type":"ContainerStarted","Data":"0f2ee5676377e40869d0d19d2937d3826ca6e2797abed55f2870b13f044da0a3"} Jul 15 12:25:07.786206 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.786182 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9hqp" event={"ID":"d23692c8-7189-494b-aa94-15feeeda6119","Type":"ContainerStarted","Data":"8a22163ec9af254142a2ff96027484a4246543e6991a92d014679a4c6cfc3ca3"} Jul 15 12:25:07.789038 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.789009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" event={"ID":"dc63c38d-1ab0-4386-89a2-11cda137d06c","Type":"ContainerStarted","Data":"b8502486ee4e19b4be9abb7780aa578515458d604a68250f0bae8e965e54f5e0"} Jul 15 12:25:07.789157 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.789138 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:07.790485 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.790463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" event={"ID":"3fa1abfd-d881-4cfe-80ce-b6892e879c84","Type":"ContainerStarted","Data":"9bd41a188e8dbca8e8f8dd9ddef84334332225c6cd631e80c94552cbc77e3aa1"} Jul 15 12:25:07.790897 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.790879 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:07.795483 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.795466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" Jul 15 12:25:07.807931 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.807887 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" podStartSLOduration=2.956281749 podStartE2EDuration="4.80786964s" totalImagesPullingTime="1.851587891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:07.806902362 +0000 UTC m=+67.019648873" watchObservedRunningTime="2026-07-15 12:25:07.80786964 +0000 UTC m=+67.020616149" Jul 15 12:25:07.835215 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.835161 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" podStartSLOduration=3.053945558 podStartE2EDuration="5.835140073s" totalImagesPullingTime="2.781194515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:07.833403882 +0000 UTC m=+67.046150393" watchObservedRunningTime="2026-07-15 12:25:07.835140073 +0000 UTC m=+67.047886584" Jul 15 12:25:07.852768 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:07.852721 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mkhdf" podStartSLOduration=1.9636707009999999 podStartE2EDuration="3.852706675s" totalImagesPullingTime="1.889035974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:07.852389095 +0000 UTC m=+67.065135605" watchObservedRunningTime="2026-07-15 12:25:07.852706675 +0000 UTC m=+67.065453185" Jul 15 12:25:09.805072 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:09.804646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvnfn" event={"ID":"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a","Type":"ContainerStarted","Data":"8afd7e2a052da906ed8b3506e666d1467d6b7ed14452bc29109ea261437320b9"} Jul 15 12:25:09.805072 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:09.804719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvnfn" event={"ID":"d1bac588-12e5-410d-8ea4-cb4ec9bbca1a","Type":"ContainerStarted","Data":"131584d879cc6379de66b6d433a8bb825848086791e7df38ea89544f5bdec1d0"} Jul 15 12:25:09.829475 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:09.829414 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gvnfn" podStartSLOduration=67.611568566 podStartE2EDuration="1m8.829395615s" totalImagesPullingTime="1.217827049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:09.827287175 +0000 UTC m=+69.040033687" watchObservedRunningTime="2026-07-15 12:25:09.829395615 +0000 UTC m=+69.042142127" Jul 15 12:25:13.813641 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.813299 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b6b6f9d9c-59h2l" Jul 15 12:25:13.830164 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.830131 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"141d9fe93dbacfeb771ce59516be73ab01e19e3a1cc475eba9d9ca093afae4c9"} Jul 15 12:25:13.830266 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.830214 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"31c2622b56c243a4d79604c557076645c6d5bc2d66d1a05ac0b5590c3a2e8dd1"} Jul 15 12:25:13.831934 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.831852 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvnkk" event={"ID":"7b1b3b6a-52c4-425f-8ecb-a1af3d5a2dbc","Type":"ContainerStarted","Data":"2fb91c7024a7a1fe45a072f3e391b6a0122c20217b14e9dc9b884c309d5bc508"} Jul 15 12:25:13.832148 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.832096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:25:13.833786 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.833761 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9hqp" event={"ID":"d23692c8-7189-494b-aa94-15feeeda6119","Type":"ContainerStarted","Data":"03b885a6d0f65df2771d50f6d33bec845940cbfe592fa546db24605e6787c00e"} Jul 15 12:25:13.888848 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.888741 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x9hqp" podStartSLOduration=66.564919986 podStartE2EDuration="1m12.888723199s" totalImagesPullingTime="6.323803213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:13.887336034 +0000 UTC m=+73.100082543" watchObservedRunningTime="2026-07-15 12:25:13.888723199 +0000 UTC m=+73.101469709" Jul 15 12:25:13.917543 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:13.917501 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tvnkk" podStartSLOduration=66.907921472 podStartE2EDuration="1m12.91748425s" totalImagesPullingTime="6.009562778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:13.915677447 +0000 UTC m=+73.128423958" watchObservedRunningTime="2026-07-15 12:25:13.91748425 +0000 UTC m=+73.130230778" Jul 15 12:25:14.469589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.469490 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:14.469589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.469584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:14.473993 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.473968 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:14.845282 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.845251 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"3dabdec2f4f88c604282b14b89534843d38cab813aeae10a933ff6614e3944d8"} Jul 15 12:25:14.845773 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.845312 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"759de1356a28bfa14089731ddcfaaeb12c8b4380659e7130fd26101523ff555d"} Jul 15 12:25:14.845773 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.845342 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"bc5833572f3343986b1c00f715458c8fbdc9a0683e43c634132ad6d499772355"} Jul 15 12:25:14.845773 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.845367 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f31fc10-6a7f-4d8d-bcc8-05a08b776761","Type":"ContainerStarted","Data":"d0d464db2bfc647b28e970768fe4baede689095cea043ccad83a7cff8ba15327"} Jul 15 12:25:14.850429 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.850396 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:25:14.878154 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:14.878083 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.110989347 podStartE2EDuration="9.878063052s" totalImagesPullingTime="6.744028376s" totalInitContainerRuntime="23.045329ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:14.876999286 +0000 UTC m=+74.089745798" watchObservedRunningTime="2026-07-15 12:25:14.878063052 +0000 UTC m=+74.090809602" Jul 15 12:25:16.003806 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:16.003770 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:25:24.211331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:24.211221 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:24.211331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:24.211294 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:29.111793 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.111755 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d5d4f5d4d-kzd9g" podUID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" containerName="console" containerID="cri-o://36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196" gracePeriod=15 Jul 15 12:25:29.377657 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.377597 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:25:29.452795 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.452769 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-service-ca\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.452947 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.452805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-oauth-serving-cert\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.452947 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.452834 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-trusted-ca-bundle\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.453060 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453014 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-config\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.453117 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453073 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spbr5\" (UniqueName: \"kubernetes.io/projected/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-kube-api-access-spbr5\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.453188 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453164 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:25:29.453244 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453216 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:25:29.453286 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453166 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-oauth-config\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.453286 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453261 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:25:29.453361 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453322 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-serving-cert\") pod \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\" (UID: \"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec\") " Jul 15 12:25:29.453414 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453366 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-config" (OuterVolumeSpecName: "console-config") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:25:29.453956 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453937 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.453956 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453955 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-oauth-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.454124 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453965 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-trusted-ca-bundle\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.454124 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.453973 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.455233 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.455212 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-kube-api-access-spbr5" (OuterVolumeSpecName: "kube-api-access-spbr5") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "kube-api-access-spbr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:25:29.455233 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.455224 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:25:29.455377 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.455295 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" (UID: "ee258c14-5ccf-4feb-8cdb-8fe4f92527ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:25:29.554407 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.554383 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.554407 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.554409 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbr5\" (UniqueName: \"kubernetes.io/projected/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-kube-api-access-spbr5\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.554577 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.554422 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec-console-oauth-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:25:29.902397 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.902364 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" containerID="36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196" exitCode=0 Jul 15 12:25:29.902524 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.902404 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d4f5d4d-kzd9g" Jul 15 12:25:29.902524 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.902402 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d4f5d4d-kzd9g" event={"ID":"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec","Type":"ContainerDied","Data":"36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196"} Jul 15 12:25:29.902637 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.902539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d4f5d4d-kzd9g" event={"ID":"ee258c14-5ccf-4feb-8cdb-8fe4f92527ec","Type":"ContainerDied","Data":"bff06b5a12be2724f86e47ddd2560342d75e9a6b04de766bef202b8172103b5a"} Jul 15 12:25:29.902637 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.902595 2582 scope.go:117] "RemoveContainer" containerID="36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196" Jul 15 12:25:29.914589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.914549 2582 scope.go:117] "RemoveContainer" containerID="36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196" Jul 15 12:25:29.914842 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:25:29.914823 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196\": container with ID starting with 36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196 not found: ID does not exist" containerID="36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196" Jul 15 12:25:29.914899 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.914849 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196"} err="failed to get container status \"36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196\": rpc error: code = NotFound desc = could not find container \"36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196\": container with ID starting with 36c6f371e61d62bcc7785671854468f31a0f12352ecbe9be626682b60a090196 not found: ID does not exist" Jul 15 12:25:29.927031 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.927012 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5d4f5d4d-kzd9g"] Jul 15 12:25:29.930846 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:29.930830 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5d4f5d4d-kzd9g"] Jul 15 12:25:31.314816 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:31.314775 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" path="/var/lib/kubelet/pods/ee258c14-5ccf-4feb-8cdb-8fe4f92527ec/volumes" Jul 15 12:25:44.216098 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:44.216068 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:44.219895 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:44.219871 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bc55c645c-qt8fh" Jul 15 12:25:44.847981 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:25:44.847950 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tvnkk" Jul 15 12:26:06.003844 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:06.003795 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:26:06.023386 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:06.023358 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:26:06.070294 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:06.070266 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:26:24.279459 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.279428 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-766c6b5949-xpqw5"] Jul 15 12:26:24.279919 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.279762 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" containerName="console" Jul 15 12:26:24.279919 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.279774 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" containerName="console" Jul 15 12:26:24.279919 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.279848 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee258c14-5ccf-4feb-8cdb-8fe4f92527ec" containerName="console" Jul 15 12:26:24.282972 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.282953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.287533 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.287509 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 12:26:24.287737 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.287717 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 12:26:24.288028 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.288009 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 12:26:24.288105 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.288051 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x7v7k\"" Jul 15 12:26:24.288160 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.288111 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 12:26:24.289542 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.289523 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 12:26:24.295293 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295388 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-telemeter-client-tls\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295388 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27d7j\" (UniqueName: \"kubernetes.io/projected/3395992c-8a59-4e37-8de7-6d760bf93d7f-kube-api-access-27d7j\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295467 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-serving-certs-ca-bundle\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295467 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-secret-telemeter-client\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295467 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295591 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-federate-client-tls\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295591 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-metrics-client-ca\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.295591 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.295548 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 12:26:24.307508 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.307488 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766c6b5949-xpqw5"] Jul 15 12:26:24.396747 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.396717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-serving-certs-ca-bundle\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.396865 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.396752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-secret-telemeter-client\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.396865 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.396776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.396993 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.396971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-federate-client-tls\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.397052 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.397036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-metrics-client-ca\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.397129 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.397111 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.397207 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.397192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-telemeter-client-tls\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.397274 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.397220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27d7j\" (UniqueName: \"kubernetes.io/projected/3395992c-8a59-4e37-8de7-6d760bf93d7f-kube-api-access-27d7j\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.397477 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.397454 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-serving-certs-ca-bundle\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.398032 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.398007 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.398126 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.398030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3395992c-8a59-4e37-8de7-6d760bf93d7f-metrics-client-ca\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.399245 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.399225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-secret-telemeter-client\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.399749 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.399726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.399822 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.399779 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-federate-client-tls\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.399872 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.399851 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3395992c-8a59-4e37-8de7-6d760bf93d7f-telemeter-client-tls\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.416616 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.416589 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27d7j\" (UniqueName: \"kubernetes.io/projected/3395992c-8a59-4e37-8de7-6d760bf93d7f-kube-api-access-27d7j\") pod \"telemeter-client-766c6b5949-xpqw5\" (UID: \"3395992c-8a59-4e37-8de7-6d760bf93d7f\") " pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.592345 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.592286 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" Jul 15 12:26:24.733225 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:24.733203 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766c6b5949-xpqw5"] Jul 15 12:26:24.735380 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:26:24.735348 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3395992c_8a59_4e37_8de7_6d760bf93d7f.slice/crio-ab046a26ee91e89b0ddfbc8b5104f00a89545bad9ece03682259ddc7cad1e118 WatchSource:0}: Error finding container ab046a26ee91e89b0ddfbc8b5104f00a89545bad9ece03682259ddc7cad1e118: Status 404 returned error can't find the container with id ab046a26ee91e89b0ddfbc8b5104f00a89545bad9ece03682259ddc7cad1e118 Jul 15 12:26:25.129977 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:25.129941 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" event={"ID":"3395992c-8a59-4e37-8de7-6d760bf93d7f","Type":"ContainerStarted","Data":"ab046a26ee91e89b0ddfbc8b5104f00a89545bad9ece03682259ddc7cad1e118"} Jul 15 12:26:27.139501 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:27.139473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" event={"ID":"3395992c-8a59-4e37-8de7-6d760bf93d7f","Type":"ContainerStarted","Data":"8cfcef7a0f53ce219e0306e56e8cae4be6bfabf66aa8f17c69a78fd41928d5ea"} Jul 15 12:26:27.139893 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:27.139528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" event={"ID":"3395992c-8a59-4e37-8de7-6d760bf93d7f","Type":"ContainerStarted","Data":"5fd8bf6b418f3f4fa351e6adfa909bc86ccbd32ef8d180e5e87d330eca42ec2d"} Jul 15 12:26:27.139893 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:27.139547 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" event={"ID":"3395992c-8a59-4e37-8de7-6d760bf93d7f","Type":"ContainerStarted","Data":"417fe2b4ac131829dc46d5b3e6641911ca1c90af2f69539b67fb1e0938cf1554"} Jul 15 12:26:27.172471 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:27.172428 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-766c6b5949-xpqw5" podStartSLOduration=1.329389097 podStartE2EDuration="3.172410523s" totalImagesPullingTime="1.843021426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:26:27.169753648 +0000 UTC m=+146.382500158" watchObservedRunningTime="2026-07-15 12:26:27.172410523 +0000 UTC m=+146.385157034" Jul 15 12:26:45.157378 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:26:45.157346 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9bbff84c-27p2j"] Jul 15 12:27:10.182799 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.182699 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f9bbff84c-27p2j" podUID="da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" containerName="console" containerID="cri-o://2a9611b3d3fac43dd39aca151d66ce7b36440520a6c638b4622baab53e57180e" gracePeriod=15 Jul 15 12:27:10.305066 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.305037 2582 generic.go:356] "Generic (PLEG): container finished" podID="da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" containerID="2a9611b3d3fac43dd39aca151d66ce7b36440520a6c638b4622baab53e57180e" exitCode=0 Jul 15 12:27:10.305185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.305084 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9bbff84c-27p2j" event={"ID":"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb","Type":"ContainerDied","Data":"2a9611b3d3fac43dd39aca151d66ce7b36440520a6c638b4622baab53e57180e"} Jul 15 12:27:10.422140 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.422119 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:27:10.476389 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476335 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-oauth-serving-cert\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476389 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476378 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-config\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476541 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-oauth-config\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476541 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476448 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gncjj\" (UniqueName: \"kubernetes.io/projected/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-kube-api-access-gncjj\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476541 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476490 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-serving-cert\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476541 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476522 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-trusted-ca-bundle\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476754 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476544 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-service-ca\") pod \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\" (UID: \"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb\") " Jul 15 12:27:10.476754 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476716 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:10.476861 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476844 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-oauth-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:10.476914 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476856 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-config" (OuterVolumeSpecName: "console-config") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:10.476961 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476931 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:10.477009 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.476961 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:10.478603 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.478579 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-kube-api-access-gncjj" (OuterVolumeSpecName: "kube-api-access-gncjj") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "kube-api-access-gncjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:27:10.478703 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.478589 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:27:10.478703 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.478653 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" (UID: "da51b7c0-dca9-46d1-a66b-f3c8c5a338bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:27:10.577321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.577293 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-oauth-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:10.577321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.577316 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncjj\" (UniqueName: \"kubernetes.io/projected/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-kube-api-access-gncjj\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:10.577470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.577327 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:10.577470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.577336 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-trusted-ca-bundle\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:10.577470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.577347 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:10.577470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:10.577355 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb-console-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:27:11.311938 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:11.311910 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9bbff84c-27p2j" Jul 15 12:27:11.314384 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:11.314360 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9bbff84c-27p2j" event={"ID":"da51b7c0-dca9-46d1-a66b-f3c8c5a338bb","Type":"ContainerDied","Data":"9f5353d815a33d48aef68469a597adce10295bdc8cb917ef0395fbfdb7f9628f"} Jul 15 12:27:11.314497 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:11.314405 2582 scope.go:117] "RemoveContainer" containerID="2a9611b3d3fac43dd39aca151d66ce7b36440520a6c638b4622baab53e57180e" Jul 15 12:27:11.337490 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:11.337465 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9bbff84c-27p2j"] Jul 15 12:27:11.341918 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:11.341899 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f9bbff84c-27p2j"] Jul 15 12:27:13.313233 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:27:13.313201 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" path="/var/lib/kubelet/pods/da51b7c0-dca9-46d1-a66b-f3c8c5a338bb/volumes" Jul 15 12:28:05.229877 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.229846 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6hrfc"] Jul 15 12:28:05.230257 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.230178 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" containerName="console" Jul 15 12:28:05.230257 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.230189 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" containerName="console" Jul 15 12:28:05.230257 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.230250 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da51b7c0-dca9-46d1-a66b-f3c8c5a338bb" containerName="console" Jul 15 12:28:05.234311 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.234292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.238536 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.238516 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 12:28:05.238664 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.238544 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 12:28:05.240234 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.239988 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9vg8v\"" Jul 15 12:28:05.245784 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.245761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6hrfc"] Jul 15 12:28:05.421746 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.421724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb7648ba-6de3-419c-8053-3dec67105f68-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6hrfc\" (UID: \"cb7648ba-6de3-419c-8053-3dec67105f68\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.421889 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.421764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ghc\" (UniqueName: \"kubernetes.io/projected/cb7648ba-6de3-419c-8053-3dec67105f68-kube-api-access-f2ghc\") pod \"cert-manager-webhook-6486ffd997-6hrfc\" (UID: \"cb7648ba-6de3-419c-8053-3dec67105f68\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.522797 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.522775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ghc\" (UniqueName: \"kubernetes.io/projected/cb7648ba-6de3-419c-8053-3dec67105f68-kube-api-access-f2ghc\") pod \"cert-manager-webhook-6486ffd997-6hrfc\" (UID: \"cb7648ba-6de3-419c-8053-3dec67105f68\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.522926 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.522835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb7648ba-6de3-419c-8053-3dec67105f68-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6hrfc\" (UID: \"cb7648ba-6de3-419c-8053-3dec67105f68\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.546244 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.546219 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb7648ba-6de3-419c-8053-3dec67105f68-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6hrfc\" (UID: \"cb7648ba-6de3-419c-8053-3dec67105f68\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.546521 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.546499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ghc\" (UniqueName: \"kubernetes.io/projected/cb7648ba-6de3-419c-8053-3dec67105f68-kube-api-access-f2ghc\") pod \"cert-manager-webhook-6486ffd997-6hrfc\" (UID: \"cb7648ba-6de3-419c-8053-3dec67105f68\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.555330 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.555310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:05.690295 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:05.690270 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6hrfc"] Jul 15 12:28:05.692066 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:28:05.692041 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb7648ba_6de3_419c_8053_3dec67105f68.slice/crio-a4024bcdcc5c3f75fedd82a492cde3a61d055b52cf77b05219bdb66e0256fef7 WatchSource:0}: Error finding container a4024bcdcc5c3f75fedd82a492cde3a61d055b52cf77b05219bdb66e0256fef7: Status 404 returned error can't find the container with id a4024bcdcc5c3f75fedd82a492cde3a61d055b52cf77b05219bdb66e0256fef7 Jul 15 12:28:06.518133 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:06.518064 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" event={"ID":"cb7648ba-6de3-419c-8053-3dec67105f68","Type":"ContainerStarted","Data":"a4024bcdcc5c3f75fedd82a492cde3a61d055b52cf77b05219bdb66e0256fef7"} Jul 15 12:28:07.868360 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:07.868321 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-78xsf"] Jul 15 12:28:07.875778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:07.875758 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:07.880533 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:07.880519 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6wsqv\"" Jul 15 12:28:07.900541 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:07.900507 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-78xsf"] Jul 15 12:28:07.941084 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:07.941054 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k4p\" (UniqueName: \"kubernetes.io/projected/27f94b5a-f510-4840-9ee6-2eb64995c1e5-kube-api-access-k2k4p\") pod \"cert-manager-cainjector-559cdd8c87-78xsf\" (UID: \"27f94b5a-f510-4840-9ee6-2eb64995c1e5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:07.941211 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:07.941107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27f94b5a-f510-4840-9ee6-2eb64995c1e5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-78xsf\" (UID: \"27f94b5a-f510-4840-9ee6-2eb64995c1e5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:08.041790 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:08.041749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k4p\" (UniqueName: \"kubernetes.io/projected/27f94b5a-f510-4840-9ee6-2eb64995c1e5-kube-api-access-k2k4p\") pod \"cert-manager-cainjector-559cdd8c87-78xsf\" (UID: \"27f94b5a-f510-4840-9ee6-2eb64995c1e5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:08.041965 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:08.041808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27f94b5a-f510-4840-9ee6-2eb64995c1e5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-78xsf\" (UID: \"27f94b5a-f510-4840-9ee6-2eb64995c1e5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:08.054142 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:08.054113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k4p\" (UniqueName: \"kubernetes.io/projected/27f94b5a-f510-4840-9ee6-2eb64995c1e5-kube-api-access-k2k4p\") pod \"cert-manager-cainjector-559cdd8c87-78xsf\" (UID: \"27f94b5a-f510-4840-9ee6-2eb64995c1e5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:08.056531 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:08.056508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27f94b5a-f510-4840-9ee6-2eb64995c1e5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-78xsf\" (UID: \"27f94b5a-f510-4840-9ee6-2eb64995c1e5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:08.184670 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:08.184583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" Jul 15 12:28:08.892756 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:08.892711 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-78xsf"] Jul 15 12:28:08.894765 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:28:08.894742 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f94b5a_f510_4840_9ee6_2eb64995c1e5.slice/crio-dc3c86c7a250f5114d9c0bff798b8edaf9a46653c51354a6f01ebc63b67fd92d WatchSource:0}: Error finding container dc3c86c7a250f5114d9c0bff798b8edaf9a46653c51354a6f01ebc63b67fd92d: Status 404 returned error can't find the container with id dc3c86c7a250f5114d9c0bff798b8edaf9a46653c51354a6f01ebc63b67fd92d Jul 15 12:28:09.534793 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:09.534751 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" event={"ID":"27f94b5a-f510-4840-9ee6-2eb64995c1e5","Type":"ContainerStarted","Data":"fb8c5dd3299b4735f31b36970a68e449fbc1f0d518b532a7702991e5f47c6d4a"} Jul 15 12:28:09.534793 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:09.534806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" event={"ID":"27f94b5a-f510-4840-9ee6-2eb64995c1e5","Type":"ContainerStarted","Data":"dc3c86c7a250f5114d9c0bff798b8edaf9a46653c51354a6f01ebc63b67fd92d"} Jul 15 12:28:09.536543 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:09.536508 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" event={"ID":"cb7648ba-6de3-419c-8053-3dec67105f68","Type":"ContainerStarted","Data":"26025ab5c32e896315e30665d2444c468fc5cde7fa1ee9aa3cf2fd6c68432084"} Jul 15 12:28:09.536702 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:09.536584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:09.564053 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:09.564008 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78xsf" podStartSLOduration=2.563991878 podStartE2EDuration="2.563991878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:09.562427403 +0000 UTC m=+248.775173912" watchObservedRunningTime="2026-07-15 12:28:09.563991878 +0000 UTC m=+248.776738389" Jul 15 12:28:09.600809 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:09.600758 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" podStartSLOduration=1.472453352 podStartE2EDuration="4.600745687s" totalImagesPullingTime="3.128292335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:09.598003762 +0000 UTC m=+248.810750283" watchObservedRunningTime="2026-07-15 12:28:09.600745687 +0000 UTC m=+248.813492196" Jul 15 12:28:15.542755 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:15.542725 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6hrfc" Jul 15 12:28:35.212387 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.212315 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl"] Jul 15 12:28:35.215733 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.215717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.221079 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.221057 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 12:28:35.221270 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.221251 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 12:28:35.221492 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.221474 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vq7wd\"" Jul 15 12:28:35.222004 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.221984 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 12:28:35.222102 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.222026 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 12:28:35.245259 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.245231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl"] Jul 15 12:28:35.251596 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.251553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/adb0a621-34a8-4b38-a01d-9571cbffbaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.251699 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.251604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/adb0a621-34a8-4b38-a01d-9571cbffbaf6-manifests\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.251699 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.251622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adb0a621-34a8-4b38-a01d-9571cbffbaf6-tmp\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.251699 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.251674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhfx\" (UniqueName: \"kubernetes.io/projected/adb0a621-34a8-4b38-a01d-9571cbffbaf6-kube-api-access-kdhfx\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.251807 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.251705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adb0a621-34a8-4b38-a01d-9571cbffbaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.352871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.352845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/adb0a621-34a8-4b38-a01d-9571cbffbaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.352871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.352874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/adb0a621-34a8-4b38-a01d-9571cbffbaf6-manifests\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.353037 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.352890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adb0a621-34a8-4b38-a01d-9571cbffbaf6-tmp\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.353037 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.352927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhfx\" (UniqueName: \"kubernetes.io/projected/adb0a621-34a8-4b38-a01d-9571cbffbaf6-kube-api-access-kdhfx\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.353037 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.352972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adb0a621-34a8-4b38-a01d-9571cbffbaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.353298 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.353272 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/adb0a621-34a8-4b38-a01d-9571cbffbaf6-manifests\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.355252 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.355225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adb0a621-34a8-4b38-a01d-9571cbffbaf6-tmp\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.355341 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.355261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/adb0a621-34a8-4b38-a01d-9571cbffbaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.355341 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.355300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adb0a621-34a8-4b38-a01d-9571cbffbaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.362518 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.362499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhfx\" (UniqueName: \"kubernetes.io/projected/adb0a621-34a8-4b38-a01d-9571cbffbaf6-kube-api-access-kdhfx\") pod \"opendatahub-operator-controller-manager-654f68fd79-tspzl\" (UID: \"adb0a621-34a8-4b38-a01d-9571cbffbaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.526294 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.526268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:35.712767 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:35.712734 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl"] Jul 15 12:28:35.721358 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:28:35.721337 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb0a621_34a8_4b38_a01d_9571cbffbaf6.slice/crio-5b142fc4b90f62d7ce559014de1407355e266c26322b2a87c1c7b8ae188398c8 WatchSource:0}: Error finding container 5b142fc4b90f62d7ce559014de1407355e266c26322b2a87c1c7b8ae188398c8: Status 404 returned error can't find the container with id 5b142fc4b90f62d7ce559014de1407355e266c26322b2a87c1c7b8ae188398c8 Jul 15 12:28:36.644545 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:36.644504 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" event={"ID":"adb0a621-34a8-4b38-a01d-9571cbffbaf6","Type":"ContainerStarted","Data":"5b142fc4b90f62d7ce559014de1407355e266c26322b2a87c1c7b8ae188398c8"} Jul 15 12:28:39.660371 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:39.660336 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" event={"ID":"adb0a621-34a8-4b38-a01d-9571cbffbaf6","Type":"ContainerStarted","Data":"03c89c75b751eb5059742142e1a1de003e7d818f6b3acaab19e2253942a0e37d"} Jul 15 12:28:40.671380 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:40.671340 2582 generic.go:356] "Generic (PLEG): container finished" podID="adb0a621-34a8-4b38-a01d-9571cbffbaf6" containerID="03c89c75b751eb5059742142e1a1de003e7d818f6b3acaab19e2253942a0e37d" exitCode=0 Jul 15 12:28:40.671861 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:40.671402 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" event={"ID":"adb0a621-34a8-4b38-a01d-9571cbffbaf6","Type":"ContainerDied","Data":"03c89c75b751eb5059742142e1a1de003e7d818f6b3acaab19e2253942a0e37d"} Jul 15 12:28:41.679411 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:41.679369 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" event={"ID":"adb0a621-34a8-4b38-a01d-9571cbffbaf6","Type":"ContainerStarted","Data":"526510e79aafd4d0efe4cfa66e6bd52037872c83f14cb9e200596f94209c625d"} Jul 15 12:28:41.679833 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:41.679531 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:41.705991 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:41.705940 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" podStartSLOduration=2.082815636 podStartE2EDuration="6.705922981s" totalImagesPullingTime="3.543256791s" totalInitContainerRuntime="1.079850554s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:41.703393716 +0000 UTC m=+280.916140230" watchObservedRunningTime="2026-07-15 12:28:41.705922981 +0000 UTC m=+280.918669491" Jul 15 12:28:52.686000 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:52.685969 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-654f68fd79-tspzl" Jul 15 12:28:55.574303 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.574269 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz"] Jul 15 12:28:55.581920 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.581887 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.586503 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.586463 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 12:28:55.586719 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.586525 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 12:28:55.590778 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.590750 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:28:55.590896 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.590784 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bc66d\"" Jul 15 12:28:55.590896 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.590814 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 12:28:55.590896 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.590828 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 12:28:55.604279 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.604246 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz"] Jul 15 12:28:55.733235 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.733202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/929be5b8-630d-4a30-a065-3e89fee67d3a-metrics-cert\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.733405 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.733270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/929be5b8-630d-4a30-a065-3e89fee67d3a-cert\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.733405 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.733312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/929be5b8-630d-4a30-a065-3e89fee67d3a-manager-config\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.733405 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.733342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtr5\" (UniqueName: \"kubernetes.io/projected/929be5b8-630d-4a30-a065-3e89fee67d3a-kube-api-access-mhtr5\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.834794 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.834711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/929be5b8-630d-4a30-a065-3e89fee67d3a-metrics-cert\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.834794 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.834776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/929be5b8-630d-4a30-a065-3e89fee67d3a-cert\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.835006 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.834818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/929be5b8-630d-4a30-a065-3e89fee67d3a-manager-config\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.835006 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.834840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtr5\" (UniqueName: \"kubernetes.io/projected/929be5b8-630d-4a30-a065-3e89fee67d3a-kube-api-access-mhtr5\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.835503 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.835479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/929be5b8-630d-4a30-a065-3e89fee67d3a-manager-config\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.837213 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.837194 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/929be5b8-630d-4a30-a065-3e89fee67d3a-cert\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.837306 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.837227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/929be5b8-630d-4a30-a065-3e89fee67d3a-metrics-cert\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.852316 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.852289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtr5\" (UniqueName: \"kubernetes.io/projected/929be5b8-630d-4a30-a065-3e89fee67d3a-kube-api-access-mhtr5\") pod \"lws-controller-manager-849c5974c6-xbvnz\" (UID: \"929be5b8-630d-4a30-a065-3e89fee67d3a\") " pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:55.893216 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:55.893192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:56.042969 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:56.042934 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz"] Jul 15 12:28:56.045114 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:28:56.045089 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929be5b8_630d_4a30_a065_3e89fee67d3a.slice/crio-38911e79c8389f300f8bbf4df084dacd0dcdb105a11225a45fdb12a92e62a3b1 WatchSource:0}: Error finding container 38911e79c8389f300f8bbf4df084dacd0dcdb105a11225a45fdb12a92e62a3b1: Status 404 returned error can't find the container with id 38911e79c8389f300f8bbf4df084dacd0dcdb105a11225a45fdb12a92e62a3b1 Jul 15 12:28:56.747497 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:56.747463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" event={"ID":"929be5b8-630d-4a30-a065-3e89fee67d3a","Type":"ContainerStarted","Data":"38911e79c8389f300f8bbf4df084dacd0dcdb105a11225a45fdb12a92e62a3b1"} Jul 15 12:28:57.715580 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.715529 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9797f867b-n5pdc"] Jul 15 12:28:57.724846 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.724825 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.731139 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.731108 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 12:28:57.733122 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.732691 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 12:28:57.733122 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.732966 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fv8gk\"" Jul 15 12:28:57.734426 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.734198 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 12:28:57.734426 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.734263 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 12:28:57.742027 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.742007 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9797f867b-n5pdc"] Jul 15 12:28:57.859246 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.859211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a3227da-368e-44f4-a26e-08bc425ed795-tmp\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.859740 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.859292 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3227da-368e-44f4-a26e-08bc425ed795-tls-certs\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.859740 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.859391 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxjd\" (UniqueName: \"kubernetes.io/projected/6a3227da-368e-44f4-a26e-08bc425ed795-kube-api-access-dxxjd\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.960051 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.960009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a3227da-368e-44f4-a26e-08bc425ed795-tmp\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.960227 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.960103 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3227da-368e-44f4-a26e-08bc425ed795-tls-certs\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.960227 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.960174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxjd\" (UniqueName: \"kubernetes.io/projected/6a3227da-368e-44f4-a26e-08bc425ed795-kube-api-access-dxxjd\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.962939 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.962907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3227da-368e-44f4-a26e-08bc425ed795-tls-certs\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.963271 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.963248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a3227da-368e-44f4-a26e-08bc425ed795-tmp\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:57.977669 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:57.977589 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxjd\" (UniqueName: \"kubernetes.io/projected/6a3227da-368e-44f4-a26e-08bc425ed795-kube-api-access-dxxjd\") pod \"kube-auth-proxy-9797f867b-n5pdc\" (UID: \"6a3227da-368e-44f4-a26e-08bc425ed795\") " pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:58.036887 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:58.036859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" Jul 15 12:28:58.760960 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:58.760922 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" event={"ID":"929be5b8-630d-4a30-a065-3e89fee67d3a","Type":"ContainerStarted","Data":"7b4490f30343dd5e3f3e362ea166b2cdec320716ad607ebfd0d5b9093d7d2698"} Jul 15 12:28:58.761156 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:58.760993 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:28:58.892939 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:28:58.892913 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3227da_368e_44f4_a26e_08bc425ed795.slice/crio-1e98e010b73cc9f4d7963c166c74375e755a5ba0dee61cee3536851adc8f3910 WatchSource:0}: Error finding container 1e98e010b73cc9f4d7963c166c74375e755a5ba0dee61cee3536851adc8f3910: Status 404 returned error can't find the container with id 1e98e010b73cc9f4d7963c166c74375e755a5ba0dee61cee3536851adc8f3910 Jul 15 12:28:58.935613 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:58.935555 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9797f867b-n5pdc"] Jul 15 12:28:59.051504 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:59.051446 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" podStartSLOduration=1.513531576 podStartE2EDuration="4.051428319s" totalImagesPullingTime="2.537896743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:59.046067795 +0000 UTC m=+298.258814306" watchObservedRunningTime="2026-07-15 12:28:59.051428319 +0000 UTC m=+298.264174829" Jul 15 12:28:59.768671 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:28:59.768601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" event={"ID":"6a3227da-368e-44f4-a26e-08bc425ed795","Type":"ContainerStarted","Data":"1e98e010b73cc9f4d7963c166c74375e755a5ba0dee61cee3536851adc8f3910"} Jul 15 12:29:01.156355 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:29:01.156317 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:29:01.160093 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:29:01.160068 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 12:29:01.160596 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:29:01.160540 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:29:01.781117 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:29:01.781046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" event={"ID":"6a3227da-368e-44f4-a26e-08bc425ed795","Type":"ContainerStarted","Data":"0a2aa893b842d3f2a6267378b02070c365c30384b615d79c9640fbdc4b435beb"} Jul 15 12:29:01.805955 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:29:01.805887 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9797f867b-n5pdc" podStartSLOduration=2.651784274 podStartE2EDuration="4.805867691s" totalImagesPullingTime="2.154083417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:29:01.805315778 +0000 UTC m=+301.018062290" watchObservedRunningTime="2026-07-15 12:29:01.805867691 +0000 UTC m=+301.018614203" Jul 15 12:29:09.771304 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:29:09.771272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-849c5974c6-xbvnz" Jul 15 12:30:53.616899 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.616859 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb"] Jul 15 12:30:53.620417 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.620392 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.624210 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.624178 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 12:30:53.624451 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.624429 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7c6ng\"" Jul 15 12:30:53.624827 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.624808 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 12:30:53.625587 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.625557 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 12:30:53.625646 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.625620 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 12:30:53.633092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.633068 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb"] Jul 15 12:30:53.767067 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.767038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2586511a-8400-4043-a1f6-9d3a6a41ccee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.767219 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.767076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9bv\" (UniqueName: \"kubernetes.io/projected/2586511a-8400-4043-a1f6-9d3a6a41ccee-kube-api-access-9h9bv\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.767219 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.767197 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2586511a-8400-4043-a1f6-9d3a6a41ccee-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.868097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.868020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2586511a-8400-4043-a1f6-9d3a6a41ccee-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.868097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.868070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2586511a-8400-4043-a1f6-9d3a6a41ccee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.868305 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.868163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9bv\" (UniqueName: \"kubernetes.io/projected/2586511a-8400-4043-a1f6-9d3a6a41ccee-kube-api-access-9h9bv\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.868305 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:30:53.868229 2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 15 12:30:53.868305 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:30:53.868291 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2586511a-8400-4043-a1f6-9d3a6a41ccee-plugin-serving-cert podName:2586511a-8400-4043-a1f6-9d3a6a41ccee nodeName:}" failed. No retries permitted until 2026-07-15 12:30:54.368276083 +0000 UTC m=+413.581022572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2586511a-8400-4043-a1f6-9d3a6a41ccee-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-g87lb" (UID: "2586511a-8400-4043-a1f6-9d3a6a41ccee") : secret "plugin-serving-cert" not found Jul 15 12:30:53.868774 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.868748 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2586511a-8400-4043-a1f6-9d3a6a41ccee-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:53.878045 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:53.878024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9bv\" (UniqueName: \"kubernetes.io/projected/2586511a-8400-4043-a1f6-9d3a6a41ccee-kube-api-access-9h9bv\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:54.372626 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:54.372584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2586511a-8400-4043-a1f6-9d3a6a41ccee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:54.375079 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:54.375047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2586511a-8400-4043-a1f6-9d3a6a41ccee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g87lb\" (UID: \"2586511a-8400-4043-a1f6-9d3a6a41ccee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:54.529599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:54.529546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" Jul 15 12:30:54.659188 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:54.659120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb"] Jul 15 12:30:54.662075 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:30:54.662046 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2586511a_8400_4043_a1f6_9d3a6a41ccee.slice/crio-845b1d187367ddbce0b7d53b63c75c6e8840376c2cfebc72e3c5e6f2ba381e18 WatchSource:0}: Error finding container 845b1d187367ddbce0b7d53b63c75c6e8840376c2cfebc72e3c5e6f2ba381e18: Status 404 returned error can't find the container with id 845b1d187367ddbce0b7d53b63c75c6e8840376c2cfebc72e3c5e6f2ba381e18 Jul 15 12:30:54.663321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:54.663304 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:30:55.290438 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:30:55.290402 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" event={"ID":"2586511a-8400-4043-a1f6-9d3a6a41ccee","Type":"ContainerStarted","Data":"845b1d187367ddbce0b7d53b63c75c6e8840376c2cfebc72e3c5e6f2ba381e18"} Jul 15 12:31:04.294978 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.294940 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6"] Jul 15 12:31:04.302437 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.302408 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:31:04.305919 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.305894 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-g466q\"" Jul 15 12:31:04.311521 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.311335 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6"] Jul 15 12:31:04.469334 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.469279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbgz\" (UniqueName: \"kubernetes.io/projected/8c4ceb92-9808-4bd0-b939-09dd7d58f0ba-kube-api-access-njbgz\") pod \"limitador-operator-controller-manager-859d7b78bd-8gqd6\" (UID: \"8c4ceb92-9808-4bd0-b939-09dd7d58f0ba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:31:04.570864 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.570779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbgz\" (UniqueName: \"kubernetes.io/projected/8c4ceb92-9808-4bd0-b939-09dd7d58f0ba-kube-api-access-njbgz\") pod \"limitador-operator-controller-manager-859d7b78bd-8gqd6\" (UID: \"8c4ceb92-9808-4bd0-b939-09dd7d58f0ba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:31:04.584359 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.584326 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbgz\" (UniqueName: \"kubernetes.io/projected/8c4ceb92-9808-4bd0-b939-09dd7d58f0ba-kube-api-access-njbgz\") pod \"limitador-operator-controller-manager-859d7b78bd-8gqd6\" (UID: \"8c4ceb92-9808-4bd0-b939-09dd7d58f0ba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:31:04.616200 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:04.616173 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:31:17.826815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:17.826785 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6"] Jul 15 12:31:17.997681 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:31:17.997617 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4ceb92_9808_4bd0_b939_09dd7d58f0ba.slice/crio-c49186457eabe081572d4100579267904d903b38ffaac988d1b03ca79d1c4ab4 WatchSource:0}: Error finding container c49186457eabe081572d4100579267904d903b38ffaac988d1b03ca79d1c4ab4: Status 404 returned error can't find the container with id c49186457eabe081572d4100579267904d903b38ffaac988d1b03ca79d1c4ab4 Jul 15 12:31:18.410727 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:18.410686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" event={"ID":"8c4ceb92-9808-4bd0-b939-09dd7d58f0ba","Type":"ContainerStarted","Data":"c49186457eabe081572d4100579267904d903b38ffaac988d1b03ca79d1c4ab4"} Jul 15 12:31:18.412005 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:18.411973 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" event={"ID":"2586511a-8400-4043-a1f6-9d3a6a41ccee","Type":"ContainerStarted","Data":"7bc940e3ffea1cbc809a20389172ad8ef1ddc3e31966f95cb3dc488861197fad"} Jul 15 12:31:18.434256 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:18.434215 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g87lb" podStartSLOduration=2.051190335 podStartE2EDuration="25.434202004s" totalImagesPullingTime="23.383011669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:31:18.432658583 +0000 UTC m=+437.645405094" watchObservedRunningTime="2026-07-15 12:31:18.434202004 +0000 UTC m=+437.646948513" Jul 15 12:31:22.432140 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:22.432115 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" event={"ID":"8c4ceb92-9808-4bd0-b939-09dd7d58f0ba","Type":"ContainerStarted","Data":"5cd848e10a025fcd3f62374c7826c9fe2d892114ade9b5941b862a8436ed7614"} Jul 15 12:31:22.432471 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:22.432224 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:31:22.455152 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:22.455111 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" podStartSLOduration=14.161536294 podStartE2EDuration="18.455098782s" totalImagesPullingTime="4.293562488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:31:22.45318645 +0000 UTC m=+441.665932959" watchObservedRunningTime="2026-07-15 12:31:22.455098782 +0000 UTC m=+441.667845291" Jul 15 12:31:33.439222 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:31:33.439147 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gqd6" Jul 15 12:32:56.757894 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.757810 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt"] Jul 15 12:32:56.761336 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.761315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.764968 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.764941 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:32:56.766405 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.766385 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:32:56.766520 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.766503 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw8fc\"" Jul 15 12:32:56.766601 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.766503 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 15 12:32:56.788272 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.788238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt"] Jul 15 12:32:56.887333 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.887288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-maas-api-tls\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.887333 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.887333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7sx\" (UniqueName: \"kubernetes.io/projected/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-kube-api-access-zs7sx\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.887545 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.887379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-openshift-service-ca\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.988693 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.988654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-openshift-service-ca\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.988872 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.988749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-maas-api-tls\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.988872 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.988771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7sx\" (UniqueName: \"kubernetes.io/projected/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-kube-api-access-zs7sx\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.989303 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.989274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-openshift-service-ca\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:56.991217 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:56.991194 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-maas-api-tls\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:57.004015 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:57.003980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7sx\" (UniqueName: \"kubernetes.io/projected/aa9e4c84-7feb-48ed-926b-4ca694fd09b2-kube-api-access-zs7sx\") pod \"maas-api-7fdf86cb69-58vwt\" (UID: \"aa9e4c84-7feb-48ed-926b-4ca694fd09b2\") " pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:57.072322 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:57.072234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:32:57.225212 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:57.225184 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt"] Jul 15 12:32:57.226147 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:32:57.226120 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9e4c84_7feb_48ed_926b_4ca694fd09b2.slice/crio-36faaabf548d13c562215a8446ee4b401128a19a70ebee93eebc4b7bcb8dc77b WatchSource:0}: Error finding container 36faaabf548d13c562215a8446ee4b401128a19a70ebee93eebc4b7bcb8dc77b: Status 404 returned error can't find the container with id 36faaabf548d13c562215a8446ee4b401128a19a70ebee93eebc4b7bcb8dc77b Jul 15 12:32:57.869136 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:32:57.869097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" event={"ID":"aa9e4c84-7feb-48ed-926b-4ca694fd09b2","Type":"ContainerStarted","Data":"36faaabf548d13c562215a8446ee4b401128a19a70ebee93eebc4b7bcb8dc77b"} Jul 15 12:33:00.886542 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:33:00.886501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" event={"ID":"aa9e4c84-7feb-48ed-926b-4ca694fd09b2","Type":"ContainerStarted","Data":"69f1efb7092497cf81d1c959b26dd83ef3969b87e5e2bff472a92774dbe61fa8"} Jul 15 12:33:00.887016 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:33:00.886738 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:33:00.918252 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:33:00.918200 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" podStartSLOduration=2.206597777 podStartE2EDuration="4.918182689s" totalImagesPullingTime="2.711584912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:33:00.914817625 +0000 UTC m=+540.127564138" watchObservedRunningTime="2026-07-15 12:33:00.918182689 +0000 UTC m=+540.130929199" Jul 15 12:33:06.897097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:33:06.897062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7fdf86cb69-58vwt" Jul 15 12:34:01.152018 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.151985 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn"] Jul 15 12:34:01.155470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.155446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.160312 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.160284 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 12:34:01.160474 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.160286 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8v9kl\"" Jul 15 12:34:01.160474 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.160340 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 12:34:01.160474 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.160341 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 12:34:01.167964 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.167942 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn"] Jul 15 12:34:01.178522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.178491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.178732 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.178545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkf8q\" (UniqueName: \"kubernetes.io/projected/129db752-348b-43f2-9bf7-e49cdca15448-kube-api-access-tkf8q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.178732 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.178608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.178732 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.178662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.178912 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.178731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.178912 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.178832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129db752-348b-43f2-9bf7-e49cdca15448-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.208867 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.208833 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:34:01.209069 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.208970 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:34:01.279520 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.279764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129db752-348b-43f2-9bf7-e49cdca15448-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.279764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.279764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkf8q\" (UniqueName: \"kubernetes.io/projected/129db752-348b-43f2-9bf7-e49cdca15448-kube-api-access-tkf8q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.279764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.279970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.279970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.280096 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.279998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.280096 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.280054 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.281970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.281947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129db752-348b-43f2-9bf7-e49cdca15448-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.282243 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.282221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129db752-348b-43f2-9bf7-e49cdca15448-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.297074 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.297042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkf8q\" (UniqueName: \"kubernetes.io/projected/129db752-348b-43f2-9bf7-e49cdca15448-kube-api-access-tkf8q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vljgn\" (UID: \"129db752-348b-43f2-9bf7-e49cdca15448\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.470144 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.470052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:01.612766 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:01.612703 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn"] Jul 15 12:34:01.615028 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:34:01.615005 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129db752_348b_43f2_9bf7_e49cdca15448.slice/crio-0f6cde2670997c567fbde9af9b0a7fc4a674112f0f6e8e4af35deca3b04eb678 WatchSource:0}: Error finding container 0f6cde2670997c567fbde9af9b0a7fc4a674112f0f6e8e4af35deca3b04eb678: Status 404 returned error can't find the container with id 0f6cde2670997c567fbde9af9b0a7fc4a674112f0f6e8e4af35deca3b04eb678 Jul 15 12:34:02.180630 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:02.180583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" event={"ID":"129db752-348b-43f2-9bf7-e49cdca15448","Type":"ContainerStarted","Data":"0f6cde2670997c567fbde9af9b0a7fc4a674112f0f6e8e4af35deca3b04eb678"} Jul 15 12:34:03.186851 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:03.186811 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" event={"ID":"129db752-348b-43f2-9bf7-e49cdca15448","Type":"ContainerStarted","Data":"66cc7f53ae8a5a1863c3d4c56f29e83a6aac76c545117e0dd941c26f62d58860"} Jul 15 12:34:03.212473 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:03.212417 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" podStartSLOduration=1.147677873 podStartE2EDuration="2.212401802s" totalImagesPullingTime="1.064723929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:34:03.211021736 +0000 UTC m=+602.423768259" watchObservedRunningTime="2026-07-15 12:34:03.212401802 +0000 UTC m=+602.425148313" Jul 15 12:34:11.471195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:11.471155 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:11.471645 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:11.471220 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:11.483665 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:11.483642 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:12.239071 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:12.239045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vljgn" Jul 15 12:34:25.788463 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.788370 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4"] Jul 15 12:34:25.794469 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.794446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:25.797769 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.797746 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 12:34:25.811433 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.811401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4"] Jul 15 12:34:25.907843 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.907813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:25.907980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.907849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:25.907980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.907903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:25.907980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.907923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f58d3d-01c1-4e83-a3d8-41645e82818e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:25.908082 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.908008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:25.908082 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:25.908039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826hb\" (UniqueName: \"kubernetes.io/projected/24f58d3d-01c1-4e83-a3d8-41645e82818e-kube-api-access-826hb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009413 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009598 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f58d3d-01c1-4e83-a3d8-41645e82818e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009598 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009598 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826hb\" (UniqueName: \"kubernetes.io/projected/24f58d3d-01c1-4e83-a3d8-41645e82818e-kube-api-access-826hb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009772 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009772 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.009963 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.010038 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.009965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.010091 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.010071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.011838 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.011807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f58d3d-01c1-4e83-a3d8-41645e82818e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.011950 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.011850 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f58d3d-01c1-4e83-a3d8-41645e82818e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.018884 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.018861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-826hb\" (UniqueName: \"kubernetes.io/projected/24f58d3d-01c1-4e83-a3d8-41645e82818e-kube-api-access-826hb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cnn4\" (UID: \"24f58d3d-01c1-4e83-a3d8-41645e82818e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.104712 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.104650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:26.241586 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.241516 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4"] Jul 15 12:34:26.245246 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:34:26.245218 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f58d3d_01c1_4e83_a3d8_41645e82818e.slice/crio-f044e983f72bd0ece42c137e4dc6b47cc2b0fc9f3c0cd0bcbfb1c653f78365ed WatchSource:0}: Error finding container f044e983f72bd0ece42c137e4dc6b47cc2b0fc9f3c0cd0bcbfb1c653f78365ed: Status 404 returned error can't find the container with id f044e983f72bd0ece42c137e4dc6b47cc2b0fc9f3c0cd0bcbfb1c653f78365ed Jul 15 12:34:26.294586 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:26.294539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" event={"ID":"24f58d3d-01c1-4e83-a3d8-41645e82818e","Type":"ContainerStarted","Data":"f044e983f72bd0ece42c137e4dc6b47cc2b0fc9f3c0cd0bcbfb1c653f78365ed"} Jul 15 12:34:27.301257 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:27.301224 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" event={"ID":"24f58d3d-01c1-4e83-a3d8-41645e82818e","Type":"ContainerStarted","Data":"1848de8792f5f5e983d466937038ce424df9565ae1f958f4b3e4eb831a5ff42f"} Jul 15 12:34:27.321753 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:27.321704 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" podStartSLOduration=2.146824264 podStartE2EDuration="2.321689019s" totalImagesPullingTime="174.864755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:34:27.320809628 +0000 UTC m=+626.533556140" watchObservedRunningTime="2026-07-15 12:34:27.321689019 +0000 UTC m=+626.534435530" Jul 15 12:34:36.104844 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:36.104798 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:36.105345 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:36.104876 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:36.117271 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:36.117243 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:34:36.352087 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:34:36.352061 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cnn4" Jul 15 12:39:01.239172 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:39:01.239130 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:39:01.241194 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:39:01.241173 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:44:01.276885 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:44:01.276853 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:44:01.278148 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:44:01.278127 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:49:01.307071 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:01.307044 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:49:01.309477 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:01.308145 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:49:14.460428 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.460398 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc"] Jul 15 12:49:14.463773 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.463756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.470901 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.470882 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:49:14.472447 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.472432 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fce6d3aa-openshift-default-dockercfg-d78rd\"" Jul 15 12:49:14.506748 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.506716 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc"] Jul 15 12:49:14.577274 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-socket\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577465 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-token\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577465 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577345 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pll4b\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-kube-api-access-pll4b\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577465 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577370 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-certs\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577465 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-credential-socket\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577465 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577459 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istiod-ca-cert\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577733 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-envoy\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577733 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-podinfo\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.577733 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.577625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-data\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678763 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-envoy\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678763 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-podinfo\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-data\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-socket\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-token\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pll4b\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-kube-api-access-pll4b\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-certs\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.678997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.678977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-credential-socket\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.679309 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.679000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istiod-ca-cert\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.679309 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.679238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-data\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.679411 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.679373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-socket\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.679466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.679446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-certs\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.679522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.679462 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-credential-socket\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.679716 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.679690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istiod-ca-cert\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.681070 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.681047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-envoy\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.681302 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.681284 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-podinfo\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.691229 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.691204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pll4b\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-kube-api-access-pll4b\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.698499 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.698444 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-token\") pod \"e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.773321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.773291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:14.934178 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.934156 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc"] Jul 15 12:49:14.936639 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:49:14.936613 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2685d4ff_c777_4391_9b02_7d1a8d0e3c28.slice/crio-bb249024344b25f9bf2c88a3e61a7826c07fd4e58d11478005e68856e20a8a46 WatchSource:0}: Error finding container bb249024344b25f9bf2c88a3e61a7826c07fd4e58d11478005e68856e20a8a46: Status 404 returned error can't find the container with id bb249024344b25f9bf2c88a3e61a7826c07fd4e58d11478005e68856e20a8a46 Jul 15 12:49:14.938534 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:14.938519 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:49:15.454611 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:15.454558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" event={"ID":"2685d4ff-c777-4391-9b02-7d1a8d0e3c28","Type":"ContainerStarted","Data":"bb249024344b25f9bf2c88a3e61a7826c07fd4e58d11478005e68856e20a8a46"} Jul 15 12:49:28.537187 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:28.537148 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:28.537466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:28.537223 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:28.537466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:28.537251 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:29.525353 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:29.525307 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" event={"ID":"2685d4ff-c777-4391-9b02-7d1a8d0e3c28","Type":"ContainerStarted","Data":"58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee"} Jul 15 12:49:29.566835 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:29.566751 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podStartSLOduration=1.968536646 podStartE2EDuration="15.566729336s" totalImagesPullingTime="13.59819269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:49:29.566300166 +0000 UTC m=+1528.779046679" watchObservedRunningTime="2026-07-15 12:49:29.566729336 +0000 UTC m=+1528.779475848" Jul 15 12:49:29.773940 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:29.773907 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:29.774119 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:29.773978 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:29.775522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:29.775448 2582 patch_prober.go:28] interesting pod/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 12:49:29.775522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:29.775494 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 12:49:30.773898 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:30.773860 2582 patch_prober.go:28] interesting pod/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 12:49:30.774254 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:30.773924 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 12:49:31.773793 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:31.773752 2582 patch_prober.go:28] interesting pod/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 12:49:31.773999 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:31.773820 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 12:49:32.774094 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:32.774059 2582 patch_prober.go:28] interesting pod/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 12:49:32.774538 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:32.774110 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 12:49:33.777880 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:33.777851 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:49:33.778935 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:49:33.778916 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:50:46.370294 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:46.370258 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc"] Jul 15 12:50:46.370801 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:46.370450 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" containerID="cri-o://58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee" gracePeriod=30 Jul 15 12:50:47.033120 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.033080 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv"] Jul 15 12:50:47.037000 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.036983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.041551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.041533 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6532cca1-gw-openshift-default-dockercfg-dxqxq\"" Jul 15 12:50:47.070126 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.070105 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv"] Jul 15 12:50:47.150987 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.150952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfx2\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-kube-api-access-zlfx2\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151129 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.150999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-credential-socket\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151129 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-certs\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151129 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-podinfo\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151251 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-token\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151251 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-data\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151251 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-socket\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151251 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-envoy\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.151251 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.151234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istiod-ca-cert\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252455 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-data\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252628 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-socket\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252628 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-envoy\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252628 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istiod-ca-cert\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252751 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfx2\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-kube-api-access-zlfx2\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252751 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-credential-socket\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252827 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-certs\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252827 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-podinfo\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252927 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-token\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252927 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252831 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-data\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.252927 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.252876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-socket\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.253179 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.253152 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-credential-socket\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.253237 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.253153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-certs\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.253386 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.253355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istiod-ca-cert\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.254984 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.254962 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-envoy\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.255227 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.255211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-podinfo\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.272425 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.272370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-token\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.272587 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.272553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfx2\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-kube-api-access-zlfx2\") pod \"e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.347459 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.347434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:47.502841 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.502571 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv"] Jul 15 12:50:47.505422 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:50:47.505390 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95131550_a69b_4c0c_a38e_5a2d26d0bc1b.slice/crio-f2ccc6f4f384907f83f0e25d485e99d9a734b46a4c45bec83b1b6b14a2fd13af WatchSource:0}: Error finding container f2ccc6f4f384907f83f0e25d485e99d9a734b46a4c45bec83b1b6b14a2fd13af: Status 404 returned error can't find the container with id f2ccc6f4f384907f83f0e25d485e99d9a734b46a4c45bec83b1b6b14a2fd13af Jul 15 12:50:47.507549 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.507519 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:47.507665 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.507602 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:47.507665 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.507636 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:47.898886 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.898784 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" event={"ID":"95131550-a69b-4c0c-a38e-5a2d26d0bc1b","Type":"ContainerStarted","Data":"092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335"} Jul 15 12:50:47.898886 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:47.898848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" event={"ID":"95131550-a69b-4c0c-a38e-5a2d26d0bc1b","Type":"ContainerStarted","Data":"f2ccc6f4f384907f83f0e25d485e99d9a734b46a4c45bec83b1b6b14a2fd13af"} Jul 15 12:50:48.347631 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.347598 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:48.352344 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.352321 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:48.386427 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.386374 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podStartSLOduration=2.386359542 podStartE2EDuration="2.386359542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:50:47.945860656 +0000 UTC m=+1607.158607178" watchObservedRunningTime="2026-07-15 12:50:48.386359542 +0000 UTC m=+1607.599106109" Jul 15 12:50:48.780680 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.780648 2582 patch_prober.go:28] interesting pod/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:50:48.781063 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.780702 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:50:48.903720 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.903691 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:48.904653 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:48.904628 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:50:51.638874 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.638467 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:50:51.801452 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801412 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istiod-ca-cert\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801452 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-socket\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801679 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801495 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-data\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801679 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-token\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801679 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801614 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pll4b\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-kube-api-access-pll4b\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801679 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801651 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-envoy\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801846 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801751 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-certs\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801846 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801805 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:50:51.801952 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801818 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-credential-socket\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.801952 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801811 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:50:51.801952 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801899 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-podinfo\") pod \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\" (UID: \"2685d4ff-c777-4391-9b02-7d1a8d0e3c28\") " Jul 15 12:50:51.802092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.801982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:50:51.802092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.802028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:50:51.802348 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.802321 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.802348 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.802348 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.802542 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.802361 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.802542 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.802376 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.802542 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.802398 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-data" (OuterVolumeSpecName: "istio-data") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:50:51.803973 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.803954 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:50:51.804070 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.804018 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-kube-api-access-pll4b" (OuterVolumeSpecName: "kube-api-access-pll4b") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "kube-api-access-pll4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:50:51.804264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.804248 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:50:51.804264 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.804255 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-token" (OuterVolumeSpecName: "istio-token") pod "2685d4ff-c777-4391-9b02-7d1a8d0e3c28" (UID: "2685d4ff-c777-4391-9b02-7d1a8d0e3c28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:50:51.903308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.903234 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.903308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.903261 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.903308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.903271 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.903308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.903279 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pll4b\" (UniqueName: \"kubernetes.io/projected/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-kube-api-access-pll4b\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.903308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.903288 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685d4ff-c777-4391-9b02-7d1a8d0e3c28-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:50:51.919354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.919320 2582 generic.go:356] "Generic (PLEG): container finished" podID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerID="58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee" exitCode=0 Jul 15 12:50:51.919480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.919360 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" Jul 15 12:50:51.919480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.919409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" event={"ID":"2685d4ff-c777-4391-9b02-7d1a8d0e3c28","Type":"ContainerDied","Data":"58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee"} Jul 15 12:50:51.919480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.919454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc" event={"ID":"2685d4ff-c777-4391-9b02-7d1a8d0e3c28","Type":"ContainerDied","Data":"bb249024344b25f9bf2c88a3e61a7826c07fd4e58d11478005e68856e20a8a46"} Jul 15 12:50:51.919480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.919475 2582 scope.go:117] "RemoveContainer" containerID="58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee" Jul 15 12:50:51.928542 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.928527 2582 scope.go:117] "RemoveContainer" containerID="58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee" Jul 15 12:50:51.928841 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:50:51.928818 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee\": container with ID starting with 58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee not found: ID does not exist" containerID="58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee" Jul 15 12:50:51.928893 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.928846 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee"} err="failed to get container status \"58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee\": rpc error: code = NotFound desc = could not find container \"58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee\": container with ID starting with 58d600ddef12cb8957e687e2c66e4663784fef1896b832a9f818c9153b7cb3ee not found: ID does not exist" Jul 15 12:50:51.945891 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.945829 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc"] Jul 15 12:50:51.950581 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:51.950537 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fce6d3aa-openshift-default-6f7ff79545-ndmqc"] Jul 15 12:50:53.314646 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:53.314554 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" path="/var/lib/kubelet/pods/2685d4ff-c777-4391-9b02-7d1a8d0e3c28/volumes" Jul 15 12:50:58.956048 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:58.955967 2582 generic.go:356] "Generic (PLEG): container finished" podID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerID="092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335" exitCode=0 Jul 15 12:50:58.956048 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:58.956004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" event={"ID":"95131550-a69b-4c0c-a38e-5a2d26d0bc1b","Type":"ContainerDied","Data":"092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335"} Jul 15 12:50:58.956462 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:58.956365 2582 scope.go:117] "RemoveContainer" containerID="092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335" Jul 15 12:50:58.957319 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:58.957293 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:58.957374 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:58.957357 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:58.957408 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:58.957391 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:59.963410 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:50:59.963374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" event={"ID":"95131550-a69b-4c0c-a38e-5a2d26d0bc1b","Type":"ContainerStarted","Data":"fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f"} Jul 15 12:51:00.349342 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:00.349296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:51:00.349551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:00.349442 2582 patch_prober.go:28] interesting pod/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:51:00.349551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:00.349501 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:51:01.348352 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:01.348310 2582 patch_prober.go:28] interesting pod/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:51:01.348798 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:01.348377 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:51:02.348305 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:02.348255 2582 patch_prober.go:28] interesting pod/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:51:02.348516 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:02.348342 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:51:03.348263 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:03.348219 2582 patch_prober.go:28] interesting pod/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:51:03.348464 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:03.348284 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:51:03.904054 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:03.904020 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:51:04.351834 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:04.351806 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:51:04.352695 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:04.352673 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:51:18.146637 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.146606 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv"] Jul 15 12:51:18.147037 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.146802 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" containerID="cri-o://fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f" gracePeriod=30 Jul 15 12:51:18.502498 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.502422 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2"] Jul 15 12:51:18.502924 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.502891 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" Jul 15 12:51:18.502924 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.502923 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" Jul 15 12:51:18.503090 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.503024 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2685d4ff-c777-4391-9b02-7d1a8d0e3c28" containerName="istio-proxy" Jul 15 12:51:18.506308 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.506290 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.509234 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.509216 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-da069d38-openshift-default-dockercfg-q8mwc\"" Jul 15 12:51:18.519218 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.519197 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2"] Jul 15 12:51:18.663362 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663334 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istiod-ca-cert\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663500 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-data\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663500 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-credential-socket\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663500 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-envoy\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663500 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-socket\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663674 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-certs\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663674 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbgq\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-kube-api-access-nmbgq\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663674 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663597 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-token\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.663674 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.663640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-podinfo\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764339 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764306 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-envoy\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764344 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-socket\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-certs\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbgq\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-kube-api-access-nmbgq\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764466 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-token\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-podinfo\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istiod-ca-cert\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764796 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-socket\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764796 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-data\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764878 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-certs\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.764938 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.764920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-credential-socket\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.765099 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.765078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-data\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.765270 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.765248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-credential-socket\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.765400 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.765385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istiod-ca-cert\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.766636 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.766617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-envoy\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.766871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.766854 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-podinfo\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.773810 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.773785 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-token\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.774067 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.774047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbgq\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-kube-api-access-nmbgq\") pod \"e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.817214 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.817190 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:18.954766 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.954739 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2"] Jul 15 12:51:18.956267 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:51:18.956235 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22045b2f_f8f8_4e05_98bf_6d5ae69dea21.slice/crio-df24b8944e565372322e1596aa74d819942e872ed3891d6de235d969188aedfb WatchSource:0}: Error finding container df24b8944e565372322e1596aa74d819942e872ed3891d6de235d969188aedfb: Status 404 returned error can't find the container with id df24b8944e565372322e1596aa74d819942e872ed3891d6de235d969188aedfb Jul 15 12:51:18.958522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.958491 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:51:18.958648 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.958548 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:51:18.958648 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:18.958607 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:51:19.058693 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:19.058612 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" event={"ID":"22045b2f-f8f8-4e05-98bf-6d5ae69dea21","Type":"ContainerStarted","Data":"7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7"} Jul 15 12:51:19.058693 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:19.058677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" event={"ID":"22045b2f-f8f8-4e05-98bf-6d5ae69dea21","Type":"ContainerStarted","Data":"df24b8944e565372322e1596aa74d819942e872ed3891d6de235d969188aedfb"} Jul 15 12:51:19.088065 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:19.087999 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" podStartSLOduration=1.087979941 podStartE2EDuration="1.087979941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:51:19.087394349 +0000 UTC m=+1638.300140861" watchObservedRunningTime="2026-07-15 12:51:19.087979941 +0000 UTC m=+1638.300726452" Jul 15 12:51:19.354496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:19.354420 2582 patch_prober.go:28] interesting pod/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 15 12:51:19.354496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:19.354475 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:51:19.817815 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:19.817766 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:20.822337 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:20.822308 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:21.068382 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:21.068349 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:21.069375 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:21.069355 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:23.412745 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.412686 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:51:23.515359 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515315 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-token\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515536 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515426 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-data\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515536 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515455 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-podinfo\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515536 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515471 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istiod-ca-cert\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515536 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515491 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlfx2\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-kube-api-access-zlfx2\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515536 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515516 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-envoy\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515848 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515545 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-certs\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515848 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515624 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-credential-socket\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515848 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515651 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-socket\") pod \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\" (UID: \"95131550-a69b-4c0c-a38e-5a2d26d0bc1b\") " Jul 15 12:51:23.515848 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515832 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:23.516055 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.515866 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:51:23.516055 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.516028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:23.516055 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.516045 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:23.516205 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.516050 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.516205 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.516084 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.516701 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.516676 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:23.517726 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.517701 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:51:23.517726 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.517717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-kube-api-access-zlfx2" (OuterVolumeSpecName: "kube-api-access-zlfx2") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "kube-api-access-zlfx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:51:23.517903 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.517886 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:51:23.517946 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.517898 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95131550-a69b-4c0c-a38e-5a2d26d0bc1b" (UID: "95131550-a69b-4c0c-a38e-5a2d26d0bc1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:23.617047 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617013 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.617047 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617041 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.617047 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617050 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlfx2\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-kube-api-access-zlfx2\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.617266 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617058 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.617266 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617069 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.617266 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617083 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:23.617266 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:23.617095 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95131550-a69b-4c0c-a38e-5a2d26d0bc1b-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:51:24.083537 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.083504 2582 generic.go:356] "Generic (PLEG): container finished" podID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerID="fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f" exitCode=0 Jul 15 12:51:24.083725 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.083551 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" Jul 15 12:51:24.083725 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.083591 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" event={"ID":"95131550-a69b-4c0c-a38e-5a2d26d0bc1b","Type":"ContainerDied","Data":"fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f"} Jul 15 12:51:24.083725 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.083646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv" event={"ID":"95131550-a69b-4c0c-a38e-5a2d26d0bc1b","Type":"ContainerDied","Data":"f2ccc6f4f384907f83f0e25d485e99d9a734b46a4c45bec83b1b6b14a2fd13af"} Jul 15 12:51:24.083725 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.083672 2582 scope.go:117] "RemoveContainer" containerID="fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f" Jul 15 12:51:24.092956 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.092940 2582 scope.go:117] "RemoveContainer" containerID="092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335" Jul 15 12:51:24.100380 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.100362 2582 scope.go:117] "RemoveContainer" containerID="fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f" Jul 15 12:51:24.100658 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:51:24.100637 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f\": container with ID starting with fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f not found: ID does not exist" containerID="fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f" Jul 15 12:51:24.100718 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.100667 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f"} err="failed to get container status \"fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f\": rpc error: code = NotFound desc = could not find container \"fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f\": container with ID starting with fbed050a1d68289647fd9ff40467243002768073cd000c5247872d4a4641f75f not found: ID does not exist" Jul 15 12:51:24.100718 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.100683 2582 scope.go:117] "RemoveContainer" containerID="092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335" Jul 15 12:51:24.100904 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:51:24.100887 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335\": container with ID starting with 092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335 not found: ID does not exist" containerID="092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335" Jul 15 12:51:24.100969 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.100914 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335"} err="failed to get container status \"092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335\": rpc error: code = NotFound desc = could not find container \"092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335\": container with ID starting with 092ea218f6c58e08bd0e18f852c9497d1828cb6fbc5e45e2c631e07d81388335 not found: ID does not exist" Jul 15 12:51:24.129839 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.129813 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv"] Jul 15 12:51:24.144332 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:24.144303 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6532cca1-gw-openshift-default-564b566c8c-xlgsv"] Jul 15 12:51:25.321331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:25.321300 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" path="/var/lib/kubelet/pods/95131550-a69b-4c0c-a38e-5a2d26d0bc1b/volumes" Jul 15 12:51:30.117158 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:30.117135 2582 generic.go:356] "Generic (PLEG): container finished" podID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerID="7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7" exitCode=0 Jul 15 12:51:30.117592 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:30.117218 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" event={"ID":"22045b2f-f8f8-4e05-98bf-6d5ae69dea21","Type":"ContainerDied","Data":"7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7"} Jul 15 12:51:30.117751 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:30.117733 2582 scope.go:117] "RemoveContainer" containerID="7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7" Jul 15 12:51:30.118765 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:30.118738 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:51:30.118858 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:30.118824 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:51:30.118922 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:30.118873 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:51:31.123990 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:31.123953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" event={"ID":"22045b2f-f8f8-4e05-98bf-6d5ae69dea21","Type":"ContainerStarted","Data":"9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032"} Jul 15 12:51:31.817790 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:31.817753 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:31.819097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:31.819073 2582 patch_prober.go:28] interesting pod/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:51:31.819213 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:31.819122 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:51:32.818513 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:32.818469 2582 patch_prober.go:28] interesting pod/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:51:32.819026 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:32.818597 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:51:33.818265 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:33.818232 2582 patch_prober.go:28] interesting pod/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:51:33.818427 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:33.818282 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:51:34.818594 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:34.818540 2582 patch_prober.go:28] interesting pod/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:51:34.818966 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:34.818608 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:51:35.821301 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:35.821274 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:35.821678 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:35.821509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:51:35.822103 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:51:35.822088 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:52:51.103172 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.103136 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2"] Jul 15 12:52:51.103736 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.103396 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" containerID="cri-o://9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032" gracePeriod=30 Jul 15 12:52:51.413368 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413276 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s"] Jul 15 12:52:51.414092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413749 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" Jul 15 12:52:51.414092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413763 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" Jul 15 12:52:51.414092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413778 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" Jul 15 12:52:51.414092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413784 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" Jul 15 12:52:51.414092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413856 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" Jul 15 12:52:51.414092 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.413867 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95131550-a69b-4c0c-a38e-5a2d26d0bc1b" containerName="istio-proxy" Jul 15 12:52:51.416970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.416954 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.420674 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.420656 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8e27a08d-openshift-default-dockercfg-xdcj7\"" Jul 15 12:52:51.432109 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.432083 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s"] Jul 15 12:52:51.506447 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwd9\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-kube-api-access-cnwd9\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506603 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-socket\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506603 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-envoy\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-data\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istiod-ca-cert\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506722 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-podinfo\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-certs\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-credential-socket\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.506871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.506785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-token\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.607684 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-certs\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.607807 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-credential-socket\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.607807 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-token\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.607915 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwd9\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-kube-api-access-cnwd9\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.607915 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-socket\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.607915 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-envoy\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608060 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-data\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608060 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.607985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istiod-ca-cert\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608060 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.608016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-podinfo\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608198 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.608056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-certs\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608198 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.608136 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-credential-socket\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608198 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.608192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-socket\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608497 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.608472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-data\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.608732 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.608713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istiod-ca-cert\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.610014 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.609989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-envoy\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.610247 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.610230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-podinfo\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.619606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.619582 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-token\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.619970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.619950 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwd9\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-kube-api-access-cnwd9\") pod \"e2e-derive-8e27a08d-openshift-default-5686559df5-k864s\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.726786 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.726729 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:51.860347 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.860324 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s"] Jul 15 12:52:51.862029 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:52:51.862007 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e18dc9_835d_416f_82f3_fbf0d8fb17ca.slice/crio-3c7e548d3413ac21cc861743f0e5a4779dd9ca78ac1258b47a98436f93095039 WatchSource:0}: Error finding container 3c7e548d3413ac21cc861743f0e5a4779dd9ca78ac1258b47a98436f93095039: Status 404 returned error can't find the container with id 3c7e548d3413ac21cc861743f0e5a4779dd9ca78ac1258b47a98436f93095039 Jul 15 12:52:51.864148 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.864120 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:52:51.864209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.864180 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:52:51.864209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:51.864205 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:52:52.513550 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:52.513513 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" event={"ID":"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca","Type":"ContainerStarted","Data":"7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62"} Jul 15 12:52:52.513935 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:52.513587 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" event={"ID":"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca","Type":"ContainerStarted","Data":"3c7e548d3413ac21cc861743f0e5a4779dd9ca78ac1258b47a98436f93095039"} Jul 15 12:52:52.539749 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:52.539689 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" podStartSLOduration=1.539676092 podStartE2EDuration="1.539676092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:52:52.538830836 +0000 UTC m=+1731.751577345" watchObservedRunningTime="2026-07-15 12:52:52.539676092 +0000 UTC m=+1731.752422602" Jul 15 12:52:52.727442 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:52.727415 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:52.731932 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:52.731907 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:53.517926 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:53.517894 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:53.518792 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:53.518771 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:52:56.388317 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.388296 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:52:56.542789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.542068 2582 generic.go:356] "Generic (PLEG): container finished" podID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerID="9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032" exitCode=0 Jul 15 12:52:56.542789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.542170 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" event={"ID":"22045b2f-f8f8-4e05-98bf-6d5ae69dea21","Type":"ContainerDied","Data":"9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032"} Jul 15 12:52:56.542789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.542224 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" event={"ID":"22045b2f-f8f8-4e05-98bf-6d5ae69dea21","Type":"ContainerDied","Data":"df24b8944e565372322e1596aa74d819942e872ed3891d6de235d969188aedfb"} Jul 15 12:52:56.542789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.542256 2582 scope.go:117] "RemoveContainer" containerID="9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032" Jul 15 12:52:56.542789 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.542426 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2" Jul 15 12:52:56.561312 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561285 2582 scope.go:117] "RemoveContainer" containerID="7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7" Jul 15 12:52:56.561752 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561725 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-token\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.561868 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561815 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-certs\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.561868 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561859 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-data\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.561977 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561898 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmbgq\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-kube-api-access-nmbgq\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.561977 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561931 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istiod-ca-cert\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.561977 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.561970 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-podinfo\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.562127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562004 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-socket\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.562127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562033 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-credential-socket\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.562127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562069 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-envoy\") pod \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\" (UID: \"22045b2f-f8f8-4e05-98bf-6d5ae69dea21\") " Jul 15 12:52:56.562270 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562230 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:56.562363 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562319 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:52:56.562441 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562373 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:56.562898 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562870 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.562898 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562890 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.562970 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.562902 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.563221 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.563197 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:56.563383 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.563307 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-data" (OuterVolumeSpecName: "istio-data") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:56.564701 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.564679 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-token" (OuterVolumeSpecName: "istio-token") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:52:56.565127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.565084 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:52:56.565484 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.565450 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:56.565648 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.565460 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-kube-api-access-nmbgq" (OuterVolumeSpecName: "kube-api-access-nmbgq") pod "22045b2f-f8f8-4e05-98bf-6d5ae69dea21" (UID: "22045b2f-f8f8-4e05-98bf-6d5ae69dea21"). InnerVolumeSpecName "kube-api-access-nmbgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:52:56.583118 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.583088 2582 scope.go:117] "RemoveContainer" containerID="9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032" Jul 15 12:52:56.583386 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:52:56.583367 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032\": container with ID starting with 9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032 not found: ID does not exist" containerID="9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032" Jul 15 12:52:56.583459 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.583399 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032"} err="failed to get container status \"9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032\": rpc error: code = NotFound desc = could not find container \"9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032\": container with ID starting with 9e9d11a388e9baa3fced129293a2c31563a9181b89e657409999652a724c6032 not found: ID does not exist" Jul 15 12:52:56.583459 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.583426 2582 scope.go:117] "RemoveContainer" containerID="7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7" Jul 15 12:52:56.583762 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:52:56.583733 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7\": container with ID starting with 7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7 not found: ID does not exist" containerID="7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7" Jul 15 12:52:56.583813 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.583769 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7"} err="failed to get container status \"7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7\": rpc error: code = NotFound desc = could not find container \"7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7\": container with ID starting with 7b369173efdd4f1f85916e3713cbe40557ec96caba5dbfcb64a21f58a6286da7 not found: ID does not exist" Jul 15 12:52:56.664209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.664130 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbgq\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-kube-api-access-nmbgq\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.664209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.664157 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.664209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.664168 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.664209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.664176 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.664209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.664184 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.664209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.664191 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22045b2f-f8f8-4e05-98bf-6d5ae69dea21-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:52:56.870191 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.870164 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2"] Jul 15 12:52:56.873929 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:56.873907 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-da069d38-openshift-default-774f455c9c-vwvk2"] Jul 15 12:52:57.315235 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:52:57.315202 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" path="/var/lib/kubelet/pods/22045b2f-f8f8-4e05-98bf-6d5ae69dea21/volumes" Jul 15 12:53:03.587797 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:03.587764 2582 generic.go:356] "Generic (PLEG): container finished" podID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerID="7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62" exitCode=0 Jul 15 12:53:03.588177 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:03.587801 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" event={"ID":"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca","Type":"ContainerDied","Data":"7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62"} Jul 15 12:53:03.588216 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:03.588192 2582 scope.go:117] "RemoveContainer" containerID="7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62" Jul 15 12:53:03.589167 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:03.589139 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:53:03.589214 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:03.589207 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:53:03.589248 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:03.589239 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:53:04.594670 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:04.594635 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" event={"ID":"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca","Type":"ContainerStarted","Data":"149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f"} Jul 15 12:53:04.727773 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:04.727736 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:53:04.728687 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:04.728665 2582 patch_prober.go:28] interesting pod/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 12:53:04.728764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:04.728712 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 12:53:05.727488 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:05.727456 2582 patch_prober.go:28] interesting pod/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 12:53:05.727873 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:05.727512 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 12:53:06.728134 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:06.728061 2582 patch_prober.go:28] interesting pod/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 12:53:06.728134 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:06.728120 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 12:53:07.727640 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:07.727606 2582 patch_prober.go:28] interesting pod/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 12:53:07.727833 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:07.727669 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 12:53:08.518653 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:08.518619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:53:08.731807 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:08.731780 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:53:08.732639 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:53:08.732618 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:54:01.342209 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:01.342180 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:54:01.346882 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:01.346861 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:54:12.322651 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:12.322622 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s"] Jul 15 12:54:12.323113 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:12.322842 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" containerID="cri-o://149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f" gracePeriod=30 Jul 15 12:54:17.572512 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.572489 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:54:17.712976 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.712945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-data\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713120 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713003 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istiod-ca-cert\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713120 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713044 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnwd9\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-kube-api-access-cnwd9\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713120 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713065 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-socket\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713120 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713089 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-credential-socket\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713324 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713179 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-envoy\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713324 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-certs\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713324 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713258 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-token\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713324 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713295 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:17.713528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713338 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-podinfo\") pod \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\" (UID: \"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca\") " Jul 15 12:54:17.713528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713381 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:17.713528 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713513 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:54:17.713740 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713719 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.713740 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:17.713740 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713738 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.713907 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713752 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.713907 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.713760 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:17.715483 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.715451 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:54:17.715483 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.715459 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:17.715749 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.715483 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:54:17.715749 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.715518 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-kube-api-access-cnwd9" (OuterVolumeSpecName: "kube-api-access-cnwd9") pod "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" (UID: "a8e18dc9-835d-416f-82f3-fbf0d8fb17ca"). InnerVolumeSpecName "kube-api-access-cnwd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:54:17.814206 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.814181 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.814206 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.814203 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwd9\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-kube-api-access-cnwd9\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.814372 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.814212 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.814372 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.814222 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.814372 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.814230 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.814372 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.814238 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:54:17.964825 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.964745 2582 generic.go:356] "Generic (PLEG): container finished" podID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerID="149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f" exitCode=0 Jul 15 12:54:17.964825 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.964793 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" Jul 15 12:54:17.964980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.964829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" event={"ID":"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca","Type":"ContainerDied","Data":"149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f"} Jul 15 12:54:17.964980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.964883 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s" event={"ID":"a8e18dc9-835d-416f-82f3-fbf0d8fb17ca","Type":"ContainerDied","Data":"3c7e548d3413ac21cc861743f0e5a4779dd9ca78ac1258b47a98436f93095039"} Jul 15 12:54:17.964980 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.964905 2582 scope.go:117] "RemoveContainer" containerID="149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f" Jul 15 12:54:17.973860 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.973846 2582 scope.go:117] "RemoveContainer" containerID="7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62" Jul 15 12:54:17.982282 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.982261 2582 scope.go:117] "RemoveContainer" containerID="149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f" Jul 15 12:54:17.982527 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:54:17.982510 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f\": container with ID starting with 149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f not found: ID does not exist" containerID="149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f" Jul 15 12:54:17.982618 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.982535 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f"} err="failed to get container status \"149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f\": rpc error: code = NotFound desc = could not find container \"149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f\": container with ID starting with 149941c90be441b9bdf452d0a1259f5e442094c5d70052a5665ad1bd93f49c9f not found: ID does not exist" Jul 15 12:54:17.982618 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.982551 2582 scope.go:117] "RemoveContainer" containerID="7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62" Jul 15 12:54:17.982812 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:54:17.982795 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62\": container with ID starting with 7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62 not found: ID does not exist" containerID="7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62" Jul 15 12:54:17.982849 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:17.982818 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62"} err="failed to get container status \"7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62\": rpc error: code = NotFound desc = could not find container \"7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62\": container with ID starting with 7d30a83229c1206c315f010674c3420238d6391060a9a9e5f8d5329e48ceec62 not found: ID does not exist" Jul 15 12:54:18.025104 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:18.025085 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s"] Jul 15 12:54:18.044945 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:18.044927 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8e27a08d-openshift-default-5686559df5-k864s"] Jul 15 12:54:19.315455 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:54:19.315423 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" path="/var/lib/kubelet/pods/a8e18dc9-835d-416f-82f3-fbf0d8fb17ca/volumes" Jul 15 12:55:52.652910 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.652880 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv"] Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653276 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653287 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653300 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653306 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653313 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653319 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653330 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" Jul 15 12:55:52.653354 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653335 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" Jul 15 12:55:52.653610 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653385 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" Jul 15 12:55:52.653610 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653396 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" Jul 15 12:55:52.653610 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.653405 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22045b2f-f8f8-4e05-98bf-6d5ae69dea21" containerName="istio-proxy" Jul 15 12:55:52.656522 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.656503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.662432 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.662405 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8635587b-openshift-default-dockercfg-mnlj8\"" Jul 15 12:55:52.662576 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.662541 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:55:52.698875 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.698847 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv"] Jul 15 12:55:52.749793 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-data\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.749944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-credential-socket\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.749944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-podinfo\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.749944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-certs\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.749944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-envoy\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.749944 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwqt\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-kube-api-access-5dwqt\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.750161 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.749987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-token\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.750161 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.750004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istiod-ca-cert\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.750161 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.750030 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-socket\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850768 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-data\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-credential-socket\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-podinfo\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-certs\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-envoy\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwqt\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-kube-api-access-5dwqt\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.850971 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-token\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851250 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.850979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istiod-ca-cert\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851250 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.851187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-credential-socket\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851250 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.851198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-socket\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851406 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.851270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-data\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851406 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.851312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-certs\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851478 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.851460 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-socket\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.851605 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.851550 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istiod-ca-cert\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.853238 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.853217 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-envoy\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.853500 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.853484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-podinfo\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.872072 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.872050 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-token\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.877455 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.877429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwqt\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-kube-api-access-5dwqt\") pod \"e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:52.967171 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:52.967095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:53.159662 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.159500 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv"] Jul 15 12:55:53.162451 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:55:53.162421 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acdd3eb_f4b0_4b34_bc27_402fe857f336.slice/crio-f5d72bb991f6389e8edfcf750df68e181b97f7e30de41fc45513e6b2eb8b86af WatchSource:0}: Error finding container f5d72bb991f6389e8edfcf750df68e181b97f7e30de41fc45513e6b2eb8b86af: Status 404 returned error can't find the container with id f5d72bb991f6389e8edfcf750df68e181b97f7e30de41fc45513e6b2eb8b86af Jul 15 12:55:53.164455 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.164426 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:55:53.164510 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.164486 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:55:53.164544 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.164512 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:55:53.410699 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.410656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" event={"ID":"8acdd3eb-f4b0-4b34-bc27-402fe857f336","Type":"ContainerStarted","Data":"6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8"} Jul 15 12:55:53.410851 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.410720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" event={"ID":"8acdd3eb-f4b0-4b34-bc27-402fe857f336","Type":"ContainerStarted","Data":"f5d72bb991f6389e8edfcf750df68e181b97f7e30de41fc45513e6b2eb8b86af"} Jul 15 12:55:53.441756 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.441710 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" podStartSLOduration=1.441693092 podStartE2EDuration="1.441693092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:55:53.439143202 +0000 UTC m=+1912.651889704" watchObservedRunningTime="2026-07-15 12:55:53.441693092 +0000 UTC m=+1912.654439603" Jul 15 12:55:53.967445 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:53.967412 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:54.972679 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:54.972644 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:55.420932 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:55.420902 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:55:55.421761 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:55:55.421741 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:09.493286 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:09.493199 2582 generic.go:356] "Generic (PLEG): container finished" podID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerID="6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8" exitCode=0 Jul 15 12:56:09.493806 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:09.493276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" event={"ID":"8acdd3eb-f4b0-4b34-bc27-402fe857f336","Type":"ContainerDied","Data":"6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8"} Jul 15 12:56:09.493866 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:09.493843 2582 scope.go:117] "RemoveContainer" containerID="6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8" Jul 15 12:56:09.494874 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:09.494845 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:56:09.494958 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:09.494912 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:56:09.494958 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:09.494944 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:56:10.421193 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:10.421152 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:10.499794 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:10.499755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" event={"ID":"8acdd3eb-f4b0-4b34-bc27-402fe857f336","Type":"ContainerStarted","Data":"ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845"} Jul 15 12:56:10.967291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:10.967254 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:10.968538 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:10.968508 2582 patch_prober.go:28] interesting pod/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:56:10.968684 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:10.968582 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:56:11.967947 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:11.967912 2582 patch_prober.go:28] interesting pod/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:56:11.968331 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:11.967983 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:56:12.967773 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:12.967734 2582 patch_prober.go:28] interesting pod/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:56:12.967961 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:12.967796 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:56:13.968260 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:13.968223 2582 patch_prober.go:28] interesting pod/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:56:13.968684 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:13.968281 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:56:14.972105 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:14.972071 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:14.972470 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:14.972335 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:14.972941 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:14.972921 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:34.344396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:34.344352 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv"] Jul 15 12:56:34.344864 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:34.344670 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" containerID="cri-o://ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845" gracePeriod=30 Jul 15 12:56:39.594014 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.593986 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:39.653606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.653500 2582 generic.go:356] "Generic (PLEG): container finished" podID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerID="ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845" exitCode=0 Jul 15 12:56:39.653606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.653543 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" event={"ID":"8acdd3eb-f4b0-4b34-bc27-402fe857f336","Type":"ContainerDied","Data":"ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845"} Jul 15 12:56:39.653606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.653592 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" Jul 15 12:56:39.653606 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.653606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv" event={"ID":"8acdd3eb-f4b0-4b34-bc27-402fe857f336","Type":"ContainerDied","Data":"f5d72bb991f6389e8edfcf750df68e181b97f7e30de41fc45513e6b2eb8b86af"} Jul 15 12:56:39.653923 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.653631 2582 scope.go:117] "RemoveContainer" containerID="ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845" Jul 15 12:56:39.662328 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.662307 2582 scope.go:117] "RemoveContainer" containerID="6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8" Jul 15 12:56:39.670725 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.670704 2582 scope.go:117] "RemoveContainer" containerID="ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845" Jul 15 12:56:39.671055 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:56:39.671032 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845\": container with ID starting with ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845 not found: ID does not exist" containerID="ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845" Jul 15 12:56:39.671141 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.671070 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845"} err="failed to get container status \"ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845\": rpc error: code = NotFound desc = could not find container \"ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845\": container with ID starting with ec51c5e35611bd4e722fb614dab370c31858d820c29068b0f046c0b602178845 not found: ID does not exist" Jul 15 12:56:39.671141 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.671097 2582 scope.go:117] "RemoveContainer" containerID="6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8" Jul 15 12:56:39.671389 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:56:39.671372 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8\": container with ID starting with 6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8 not found: ID does not exist" containerID="6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8" Jul 15 12:56:39.671431 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.671396 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8"} err="failed to get container status \"6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8\": rpc error: code = NotFound desc = could not find container \"6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8\": container with ID starting with 6ac24762c3043cfe6166b94242c64f2ef0823f9554f6a09d8123712743fd09d8 not found: ID does not exist" Jul 15 12:56:39.697908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.697871 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-envoy\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.697908 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.697910 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dwqt\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-kube-api-access-5dwqt\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.697929 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-token\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.697965 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-data\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.697990 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istiod-ca-cert\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-certs\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698114 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-socket\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698392 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698144 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-podinfo\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698392 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698185 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-credential-socket\") pod \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\" (UID: \"8acdd3eb-f4b0-4b34-bc27-402fe857f336\") " Jul 15 12:56:39.698392 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698367 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:39.698553 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698448 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:39.698654 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698600 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:39.698768 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698740 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:56:39.698901 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698779 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.698901 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.698797 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.699035 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.699016 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-data" (OuterVolumeSpecName: "istio-data") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:39.700435 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.700413 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:39.700435 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.700417 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:56:39.700656 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.700633 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-kube-api-access-5dwqt" (OuterVolumeSpecName: "kube-api-access-5dwqt") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "kube-api-access-5dwqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:56:39.700656 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.700642 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-token" (OuterVolumeSpecName: "istio-token") pod "8acdd3eb-f4b0-4b34-bc27-402fe857f336" (UID: "8acdd3eb-f4b0-4b34-bc27-402fe857f336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:56:39.799510 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799473 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.799510 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799503 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.799510 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799513 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwqt\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-kube-api-access-5dwqt\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.799510 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799523 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.799817 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799532 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.799817 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799541 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.799817 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.799549 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acdd3eb-f4b0-4b34-bc27-402fe857f336-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:56:39.979934 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.979890 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv"] Jul 15 12:56:39.987303 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:39.987263 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8635587b-openshift-default-7574cd7759-f9dwv"] Jul 15 12:56:41.315321 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:41.315287 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" path="/var/lib/kubelet/pods/8acdd3eb-f4b0-4b34-bc27-402fe857f336/volumes" Jul 15 12:56:55.208418 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.208336 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x"] Jul 15 12:56:55.208843 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.208826 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" Jul 15 12:56:55.208843 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.208844 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" Jul 15 12:56:55.208931 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.208919 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" Jul 15 12:56:55.208931 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.208930 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e18dc9-835d-416f-82f3-fbf0d8fb17ca" containerName="istio-proxy" Jul 15 12:56:55.213986 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.213968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.217333 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.217306 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:56:55.217699 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.217682 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a2ddf8f4-openshift-default-dockercfg-lm4z8\"" Jul 15 12:56:55.226954 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.226927 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x"] Jul 15 12:56:55.339276 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-podinfo\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339276 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istiod-ca-cert\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339535 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-token\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339535 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-certs\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339535 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-data\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339535 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-socket\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339768 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-envoy\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339768 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-credential-socket\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.339768 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.339642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxz9l\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-kube-api-access-qxz9l\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440235 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-token\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-certs\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-data\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440313 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-socket\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-envoy\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-credential-socket\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440665 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxz9l\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-kube-api-access-qxz9l\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440665 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-podinfo\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440665 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istiod-ca-cert\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440810 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-data\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440810 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-credential-socket\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440921 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440903 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-socket\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.440998 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.440981 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-certs\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.441253 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.441231 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istiod-ca-cert\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.442694 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.442676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-envoy\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.443103 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.443081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-podinfo\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.450067 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.450042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-token\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.450274 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.450257 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxz9l\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-kube-api-access-qxz9l\") pod \"e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.524917 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.524886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:55.670790 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.670763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x"] Jul 15 12:56:55.673483 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:56:55.673455 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1f8e85_224d_4e0a_9241_1f6c1d6a95e2.slice/crio-7310e044a8e94e1182949609c2f4f18e77e2c81bc5f0dbdab275d06b80185762 WatchSource:0}: Error finding container 7310e044a8e94e1182949609c2f4f18e77e2c81bc5f0dbdab275d06b80185762: Status 404 returned error can't find the container with id 7310e044a8e94e1182949609c2f4f18e77e2c81bc5f0dbdab275d06b80185762 Jul 15 12:56:55.676128 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.676097 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:56:55.676221 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.676192 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:56:55.676284 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.676241 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:56:55.735990 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.735961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" event={"ID":"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2","Type":"ContainerStarted","Data":"7310e044a8e94e1182949609c2f4f18e77e2c81bc5f0dbdab275d06b80185762"} Jul 15 12:56:55.760340 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:55.760286 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podStartSLOduration=0.760267301 podStartE2EDuration="760.267301ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:56:55.758339107 +0000 UTC m=+1974.971085621" watchObservedRunningTime="2026-07-15 12:56:55.760267301 +0000 UTC m=+1974.973013874" Jul 15 12:56:56.525588 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:56.525521 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:56.530444 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:56.530416 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:56.743824 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:56.743786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" event={"ID":"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2","Type":"ContainerStarted","Data":"a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583"} Jul 15 12:56:56.744098 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:56.744084 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:56:56.745183 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:56:56.745164 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:57:07.047937 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.047892 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml"] Jul 15 12:57:07.056114 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.056078 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" Jul 15 12:57:07.056333 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.056318 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" Jul 15 12:57:07.056599 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.056584 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8acdd3eb-f4b0-4b34-bc27-402fe857f336" containerName="istio-proxy" Jul 15 12:57:07.060288 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.060261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.066079 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.066043 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a2ddf8f4\"/\"kube-root-ca.crt\"" Jul 15 12:57:07.066268 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.066149 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a2ddf8f4\"/\"default-dockercfg-qqmjs\"" Jul 15 12:57:07.066415 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.066392 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a2ddf8f4\"/\"e2e-lifecycle-model-a2ddf8f4-kserve-self-signed-certs\"" Jul 15 12:57:07.067992 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.067963 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml"] Jul 15 12:57:07.069299 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.069271 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a2ddf8f4\"/\"openshift-service-ca.crt\"" Jul 15 12:57:07.163274 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.163225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wdr\" (UniqueName: \"kubernetes.io/projected/8a459094-2823-47fd-b208-dd96ae6ac0e4-kube-api-access-j4wdr\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.163274 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.163271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-tmp-dir\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.163551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.163292 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-dshm\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.163551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.163364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-home\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.163551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.163504 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a459094-2823-47fd-b208-dd96ae6ac0e4-tls-certs\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.163551 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.163528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-model-cache\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264142 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a459094-2823-47fd-b208-dd96ae6ac0e4-tls-certs\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264142 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-model-cache\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264400 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wdr\" (UniqueName: \"kubernetes.io/projected/8a459094-2823-47fd-b208-dd96ae6ac0e4-kube-api-access-j4wdr\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264400 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-tmp-dir\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264400 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-dshm\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264400 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-home\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264659 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264636 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-model-cache\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264851 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-home\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.264941 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.264872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-tmp-dir\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.266704 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.266679 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-dshm\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.266951 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.266934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a459094-2823-47fd-b208-dd96ae6ac0e4-tls-certs\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.276795 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.276766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wdr\" (UniqueName: \"kubernetes.io/projected/8a459094-2823-47fd-b208-dd96ae6ac0e4-kube-api-access-j4wdr\") pod \"e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.375129 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.375011 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:07.557156 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.557093 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml"] Jul 15 12:57:07.562018 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:57:07.561986 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a459094_2823_47fd_b208_dd96ae6ac0e4.slice/crio-45dcb07f834c8c8645ad09e574a4fa433bdd571808dba184119eb360539f1e69 WatchSource:0}: Error finding container 45dcb07f834c8c8645ad09e574a4fa433bdd571808dba184119eb360539f1e69: Status 404 returned error can't find the container with id 45dcb07f834c8c8645ad09e574a4fa433bdd571808dba184119eb360539f1e69 Jul 15 12:57:07.801640 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.801601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" event={"ID":"8a459094-2823-47fd-b208-dd96ae6ac0e4","Type":"ContainerStarted","Data":"52cc7f5858e53582b56e2410631fece185aeafd36b538565866da2cef1f0754c"} Jul 15 12:57:07.801821 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.801655 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" event={"ID":"8a459094-2823-47fd-b208-dd96ae6ac0e4","Type":"ContainerStarted","Data":"45dcb07f834c8c8645ad09e574a4fa433bdd571808dba184119eb360539f1e69"} Jul 15 12:57:07.823967 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:07.823910 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" podStartSLOduration=0.823891149 podStartE2EDuration="823.891149ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:57:07.823363875 +0000 UTC m=+1987.036110388" watchObservedRunningTime="2026-07-15 12:57:07.823891149 +0000 UTC m=+1987.036637660" Jul 15 12:57:11.816774 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:11.815639 2582 patch_prober.go:28] interesting pod/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": read tcp 10.134.0.2:34010->10.134.0.36:15021: read: connection reset by peer" start-of-body= Jul 15 12:57:11.816774 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:11.815704 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": read tcp 10.134.0.2:34010->10.134.0.36:15021: read: connection reset by peer" Jul 15 12:57:12.827156 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:12.827122 2582 generic.go:356] "Generic (PLEG): container finished" podID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerID="a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583" exitCode=0 Jul 15 12:57:12.827706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:12.827199 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" event={"ID":"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2","Type":"ContainerDied","Data":"a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583"} Jul 15 12:57:12.827706 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:12.827656 2582 scope.go:117] "RemoveContainer" containerID="a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583" Jul 15 12:57:12.828772 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:12.828741 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:57:12.828850 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:12.828815 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:57:12.828891 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:12.828851 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:57:13.837703 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:13.837665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" event={"ID":"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2","Type":"ContainerStarted","Data":"7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e"} Jul 15 12:57:14.525454 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:14.525417 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:57:14.525654 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:14.525618 2582 patch_prober.go:28] interesting pod/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:57:14.525704 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:14.525658 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:57:15.525658 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:15.525614 2582 patch_prober.go:28] interesting pod/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:57:15.526089 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:15.525684 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:57:16.525612 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:16.525541 2582 patch_prober.go:28] interesting pod/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:57:16.525798 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:16.525634 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:57:17.375844 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:17.375810 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:17.376033 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:17.375863 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:17.388238 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:17.388207 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:17.526176 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:17.526139 2582 patch_prober.go:28] interesting pod/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:57:17.526597 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:17.526196 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:57:17.875635 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:17.875607 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:57:18.529451 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:18.529421 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:57:18.529871 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:18.529747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:57:18.530365 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:18.530348 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:57:50.952195 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:57:50.952156 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml"] Jul 15 12:58:05.974036 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:05.973989 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" podUID="8a459094-2823-47fd-b208-dd96ae6ac0e4" containerName="main" containerID="cri-o://52cc7f5858e53582b56e2410631fece185aeafd36b538565866da2cef1f0754c" gracePeriod=45 Jul 15 12:58:06.108138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.108098 2582 generic.go:356] "Generic (PLEG): container finished" podID="8a459094-2823-47fd-b208-dd96ae6ac0e4" containerID="52cc7f5858e53582b56e2410631fece185aeafd36b538565866da2cef1f0754c" exitCode=0 Jul 15 12:58:06.108313 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.108163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" event={"ID":"8a459094-2823-47fd-b208-dd96ae6ac0e4","Type":"ContainerDied","Data":"52cc7f5858e53582b56e2410631fece185aeafd36b538565866da2cef1f0754c"} Jul 15 12:58:06.237585 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.237487 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:58:06.376659 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376629 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-model-cache\") pod \"8a459094-2823-47fd-b208-dd96ae6ac0e4\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " Jul 15 12:58:06.376853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376683 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-dshm\") pod \"8a459094-2823-47fd-b208-dd96ae6ac0e4\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " Jul 15 12:58:06.376853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376743 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4wdr\" (UniqueName: \"kubernetes.io/projected/8a459094-2823-47fd-b208-dd96ae6ac0e4-kube-api-access-j4wdr\") pod \"8a459094-2823-47fd-b208-dd96ae6ac0e4\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " Jul 15 12:58:06.376853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376770 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a459094-2823-47fd-b208-dd96ae6ac0e4-tls-certs\") pod \"8a459094-2823-47fd-b208-dd96ae6ac0e4\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " Jul 15 12:58:06.376853 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376796 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-tmp-dir\") pod \"8a459094-2823-47fd-b208-dd96ae6ac0e4\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " Jul 15 12:58:06.377073 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376869 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-home\") pod \"8a459094-2823-47fd-b208-dd96ae6ac0e4\" (UID: \"8a459094-2823-47fd-b208-dd96ae6ac0e4\") " Jul 15 12:58:06.377073 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.376980 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a459094-2823-47fd-b208-dd96ae6ac0e4" (UID: "8a459094-2823-47fd-b208-dd96ae6ac0e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:06.377185 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.377083 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a459094-2823-47fd-b208-dd96ae6ac0e4" (UID: "8a459094-2823-47fd-b208-dd96ae6ac0e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:06.377238 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.377175 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-home" (OuterVolumeSpecName: "home") pod "8a459094-2823-47fd-b208-dd96ae6ac0e4" (UID: "8a459094-2823-47fd-b208-dd96ae6ac0e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:06.377600 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.377578 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-home\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:06.377710 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.377605 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-model-cache\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:06.377710 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.377620 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-tmp-dir\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:06.379198 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.379172 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a459094-2823-47fd-b208-dd96ae6ac0e4-kube-api-access-j4wdr" (OuterVolumeSpecName: "kube-api-access-j4wdr") pod "8a459094-2823-47fd-b208-dd96ae6ac0e4" (UID: "8a459094-2823-47fd-b208-dd96ae6ac0e4"). InnerVolumeSpecName "kube-api-access-j4wdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:58:06.379383 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.379357 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-dshm" (OuterVolumeSpecName: "dshm") pod "8a459094-2823-47fd-b208-dd96ae6ac0e4" (UID: "8a459094-2823-47fd-b208-dd96ae6ac0e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:06.379480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.379399 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a459094-2823-47fd-b208-dd96ae6ac0e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a459094-2823-47fd-b208-dd96ae6ac0e4" (UID: "8a459094-2823-47fd-b208-dd96ae6ac0e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:58:06.478360 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.478318 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wdr\" (UniqueName: \"kubernetes.io/projected/8a459094-2823-47fd-b208-dd96ae6ac0e4-kube-api-access-j4wdr\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:06.478360 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.478354 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a459094-2823-47fd-b208-dd96ae6ac0e4-tls-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:06.478360 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:06.478365 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a459094-2823-47fd-b208-dd96ae6ac0e4-dshm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:07.114730 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:07.114689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" event={"ID":"8a459094-2823-47fd-b208-dd96ae6ac0e4","Type":"ContainerDied","Data":"45dcb07f834c8c8645ad09e574a4fa433bdd571808dba184119eb360539f1e69"} Jul 15 12:58:07.114730 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:07.114725 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml" Jul 15 12:58:07.114730 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:07.114740 2582 scope.go:117] "RemoveContainer" containerID="52cc7f5858e53582b56e2410631fece185aeafd36b538565866da2cef1f0754c" Jul 15 12:58:07.151295 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:07.151250 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml"] Jul 15 12:58:07.161653 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:07.161619 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a2ddf8f4/e2e-lifecycle-model-a2ddf8f4-kserve-59d99d8499-dpsml"] Jul 15 12:58:07.315396 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:07.315357 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a459094-2823-47fd-b208-dd96ae6ac0e4" path="/var/lib/kubelet/pods/8a459094-2823-47fd-b208-dd96ae6ac0e4/volumes" Jul 15 12:58:48.376758 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:48.376719 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x"] Jul 15 12:58:48.377311 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:48.377011 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" containerID="cri-o://7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e" gracePeriod=30 Jul 15 12:58:48.532048 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:48.532008 2582 patch_prober.go:28] interesting pod/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:58:48.532215 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:48.532069 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:58:53.642499 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.642411 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:58:53.736609 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736543 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-certs\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.736609 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736609 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-socket\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.736875 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736705 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxz9l\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-kube-api-access-qxz9l\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.736875 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736787 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istiod-ca-cert\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.736875 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736835 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:53.736875 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736854 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-envoy\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.737097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736888 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-credential-socket\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.737097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736923 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-data\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.737097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736936 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:53.737097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736950 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-token\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.737097 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.736973 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-podinfo\") pod \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\" (UID: \"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2\") " Jul 15 12:58:53.737342 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.737096 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:53.737342 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.737279 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:58:53.737342 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.737326 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.737480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.737346 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.737480 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.737361 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.737997 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.737967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:53.739481 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.739446 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:58:53.739617 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.739500 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-kube-api-access-qxz9l" (OuterVolumeSpecName: "kube-api-access-qxz9l") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "kube-api-access-qxz9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:58:53.739764 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.739745 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:58:53.739935 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.739818 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" (UID: "ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:58:53.838123 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.838075 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxz9l\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-kube-api-access-qxz9l\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.838123 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.838115 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.838123 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.838128 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.838387 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.838139 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.838387 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.838150 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:53.838387 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:53.838162 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 12:58:54.355920 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.355874 2582 generic.go:356] "Generic (PLEG): container finished" podID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerID="7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e" exitCode=0 Jul 15 12:58:54.356127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.355927 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" Jul 15 12:58:54.356127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.355968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" event={"ID":"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2","Type":"ContainerDied","Data":"7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e"} Jul 15 12:58:54.356127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.356024 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x" event={"ID":"ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2","Type":"ContainerDied","Data":"7310e044a8e94e1182949609c2f4f18e77e2c81bc5f0dbdab275d06b80185762"} Jul 15 12:58:54.356127 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.356048 2582 scope.go:117] "RemoveContainer" containerID="7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e" Jul 15 12:58:54.365198 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.365176 2582 scope.go:117] "RemoveContainer" containerID="a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583" Jul 15 12:58:54.373889 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.373864 2582 scope.go:117] "RemoveContainer" containerID="7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e" Jul 15 12:58:54.374271 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:58:54.374245 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e\": container with ID starting with 7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e not found: ID does not exist" containerID="7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e" Jul 15 12:58:54.374330 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.374281 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e"} err="failed to get container status \"7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e\": rpc error: code = NotFound desc = could not find container \"7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e\": container with ID starting with 7b027c63c3d0357fd80a749fd91e93e55ea2960b66c097c805e875c885f9313e not found: ID does not exist" Jul 15 12:58:54.374330 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.374300 2582 scope.go:117] "RemoveContainer" containerID="a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583" Jul 15 12:58:54.374625 ip-10-0-128-235 kubenswrapper[2582]: E0715 12:58:54.374606 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583\": container with ID starting with a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583 not found: ID does not exist" containerID="a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583" Jul 15 12:58:54.374705 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.374631 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583"} err="failed to get container status \"a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583\": rpc error: code = NotFound desc = could not find container \"a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583\": container with ID starting with a42a3049df166a826a98e7658bbf6c4c25e1d3fb562dbd8e28295e509f37e583 not found: ID does not exist" Jul 15 12:58:54.394235 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.394201 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x"] Jul 15 12:58:54.395898 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:54.395869 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a2ddf8f4-openshift-default-777b7f455f-s6k2x"] Jul 15 12:58:55.315476 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:58:55.315439 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" path="/var/lib/kubelet/pods/ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2/volumes" Jul 15 12:59:01.374883 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:01.374848 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:59:01.380849 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:01.380820 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 12:59:46.119835 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.119803 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m"] Jul 15 12:59:46.120292 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.120272 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a459094-2823-47fd-b208-dd96ae6ac0e4" containerName="main" Jul 15 12:59:46.120292 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.120288 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a459094-2823-47fd-b208-dd96ae6ac0e4" containerName="main" Jul 15 12:59:46.120365 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.120313 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" Jul 15 12:59:46.120365 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.120319 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" Jul 15 12:59:46.120427 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.120380 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" Jul 15 12:59:46.120427 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.120389 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a459094-2823-47fd-b208-dd96ae6ac0e4" containerName="main" Jul 15 12:59:46.123347 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.123330 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.126387 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.126359 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:59:46.126387 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.126379 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fe3a4-openshift-default-dockercfg-k5brr\"" Jul 15 12:59:46.139428 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.139401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m"] Jul 15 12:59:46.217943 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.217912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-envoy\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.217948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-certs\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.217967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-data\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.218055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-socket\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218135 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.218109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-podinfo\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218316 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.218158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-credential-socket\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218316 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.218178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdnh\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-kube-api-access-lkdnh\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218316 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.218238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def0c5a1-d7a2-4910-9480-50d49bdc0774-istiod-ca-cert\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.218316 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.218271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-token\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318708 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-credential-socket\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318708 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdnh\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-kube-api-access-lkdnh\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def0c5a1-d7a2-4910-9480-50d49bdc0774-istiod-ca-cert\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-token\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-envoy\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-certs\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-data\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-socket\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.318979 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.318965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-podinfo\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.319339 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.319142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-credential-socket\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.319339 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.319192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-certs\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.319339 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.319305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-data\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.319484 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.319456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-socket\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.319582 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.319540 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def0c5a1-d7a2-4910-9480-50d49bdc0774-istiod-ca-cert\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.321380 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.321353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-envoy\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.321498 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.321484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-podinfo\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.328274 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.328250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-token\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.328370 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.328356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdnh\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-kube-api-access-lkdnh\") pod \"e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.433450 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.433373 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:46.571226 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.571204 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m"] Jul 15 12:59:46.573406 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:59:46.573374 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef0c5a1_d7a2_4910_9480_50d49bdc0774.slice/crio-74ac8ad568264677e90b34fe014f1fee085b5371586c4af261645efe0912b554 WatchSource:0}: Error finding container 74ac8ad568264677e90b34fe014f1fee085b5371586c4af261645efe0912b554: Status 404 returned error can't find the container with id 74ac8ad568264677e90b34fe014f1fee085b5371586c4af261645efe0912b554 Jul 15 12:59:46.575414 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.575385 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:59:46.575498 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.575442 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:59:46.575498 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.575469 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:59:46.615603 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:46.615579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" event={"ID":"def0c5a1-d7a2-4910-9480-50d49bdc0774","Type":"ContainerStarted","Data":"74ac8ad568264677e90b34fe014f1fee085b5371586c4af261645efe0912b554"} Jul 15 12:59:47.622241 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:47.622206 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" event={"ID":"def0c5a1-d7a2-4910-9480-50d49bdc0774","Type":"ContainerStarted","Data":"cc474a5c0b1d7a9503f13cac17dd88dd4fd95d67f9f914dd9bb62cfae0f520cb"} Jul 15 12:59:47.647867 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:47.647815 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podStartSLOduration=1.647798383 podStartE2EDuration="1.647798383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:59:47.646798769 +0000 UTC m=+2146.859545282" watchObservedRunningTime="2026-07-15 12:59:47.647798383 +0000 UTC m=+2146.860544895" Jul 15 12:59:48.433825 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:48.433784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:48.438617 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:48.438588 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:48.627064 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:48.627031 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:48.628079 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:48.628058 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 12:59:54.810203 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.810120 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5"] Jul 15 12:59:54.810730 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.810525 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" Jul 15 12:59:54.810730 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.810541 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" Jul 15 12:59:54.810730 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.810649 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1f8e85-224d-4e0a-9241-1f6c1d6a95e2" containerName="istio-proxy" Jul 15 12:59:54.814267 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.814246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.817414 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.817392 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-34f439-openshift-default-dockercfg-9crqm\"" Jul 15 12:59:54.826120 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.826098 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5"] Jul 15 12:59:54.902942 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.902911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-istio-token\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.902942 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.902944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-credential-socket\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.902976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-envoy\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.903051 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbsc\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-kube-api-access-6zbsc\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.903093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-certs\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903138 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.903130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba612a6-9383-4c57-a11d-e77320fe8336-istiod-ca-cert\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903262 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.903200 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba612a6-9383-4c57-a11d-e77320fe8336-istio-podinfo\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903262 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.903222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-data\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:54.903327 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:54.903278 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-socket\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004288 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-socket\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004288 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-istio-token\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-credential-socket\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-envoy\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbsc\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-kube-api-access-6zbsc\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-certs\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004589 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba612a6-9383-4c57-a11d-e77320fe8336-istiod-ca-cert\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004860 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba612a6-9383-4c57-a11d-e77320fe8336-istio-podinfo\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004860 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-data\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.004860 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004711 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-socket\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.005016 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-certs\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.005016 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.004961 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-data\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.005106 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.005029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-credential-socket\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.005282 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.005263 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba612a6-9383-4c57-a11d-e77320fe8336-istiod-ca-cert\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.006799 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.006778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-envoy\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.006925 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.006907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba612a6-9383-4c57-a11d-e77320fe8336-istio-podinfo\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.014202 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.014180 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-istio-token\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.014374 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.014352 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbsc\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-kube-api-access-6zbsc\") pod \"e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.126605 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.126485 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:55.278769 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.278740 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5"] Jul 15 12:59:55.279915 ip-10-0-128-235 kubenswrapper[2582]: W0715 12:59:55.279887 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba612a6_9383_4c57_a11d_e77320fe8336.slice/crio-49fe7b838b5bc740e50c1141596fbd6a8cc9d88e148f6eb14d18af2ec9ab2815 WatchSource:0}: Error finding container 49fe7b838b5bc740e50c1141596fbd6a8cc9d88e148f6eb14d18af2ec9ab2815: Status 404 returned error can't find the container with id 49fe7b838b5bc740e50c1141596fbd6a8cc9d88e148f6eb14d18af2ec9ab2815 Jul 15 12:59:55.282189 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.282153 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:59:55.282291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.282233 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:59:55.282291 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.282279 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:59:55.662496 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.662461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" event={"ID":"bba612a6-9383-4c57-a11d-e77320fe8336","Type":"ContainerStarted","Data":"6b41b6bc4856c5522f74127b98964b3b37f72210b575790b62046a14a805fc5a"} Jul 15 12:59:55.662695 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.662511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" event={"ID":"bba612a6-9383-4c57-a11d-e77320fe8336","Type":"ContainerStarted","Data":"49fe7b838b5bc740e50c1141596fbd6a8cc9d88e148f6eb14d18af2ec9ab2815"} Jul 15 12:59:55.702965 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:55.702921 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podStartSLOduration=1.702906332 podStartE2EDuration="1.702906332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:59:55.702588305 +0000 UTC m=+2154.915334813" watchObservedRunningTime="2026-07-15 12:59:55.702906332 +0000 UTC m=+2154.915652843" Jul 15 12:59:56.127635 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:56.127602 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:56.132666 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:56.132642 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:56.666938 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:56.666900 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 12:59:56.667747 ip-10-0-128-235 kubenswrapper[2582]: I0715 12:59:56.667730 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:00:04.238532 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.238484 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:57968->10.134.0.38:15021: read: connection reset by peer" start-of-body= Jul 15 13:00:04.245787 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.239122 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:57968->10.134.0.38:15021: read: connection reset by peer" Jul 15 13:00:04.708679 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.708642 2582 generic.go:356] "Generic (PLEG): container finished" podID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerID="cc474a5c0b1d7a9503f13cac17dd88dd4fd95d67f9f914dd9bb62cfae0f520cb" exitCode=0 Jul 15 13:00:04.708869 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.708718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" event={"ID":"def0c5a1-d7a2-4910-9480-50d49bdc0774","Type":"ContainerDied","Data":"cc474a5c0b1d7a9503f13cac17dd88dd4fd95d67f9f914dd9bb62cfae0f520cb"} Jul 15 13:00:04.709158 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.709144 2582 scope.go:117] "RemoveContainer" containerID="cc474a5c0b1d7a9503f13cac17dd88dd4fd95d67f9f914dd9bb62cfae0f520cb" Jul 15 13:00:04.710136 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.710107 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:00:04.710241 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.710173 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:00:04.710241 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:04.710204 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:00:05.645001 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.644963 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb"] Jul 15 13:00:05.651331 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.650156 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.654063 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.654014 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fe3a4\"/\"default-dockercfg-v57vx\"" Jul 15 13:00:05.656171 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.655947 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fe3a4\"/\"kube-root-ca.crt\"" Jul 15 13:00:05.656936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.656458 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fe3a4\"/\"openshift-service-ca.crt\"" Jul 15 13:00:05.656936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.656780 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fe3a4\"/\"test-model-1fe3a4-kserve-self-signed-certs\"" Jul 15 13:00:05.666398 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.666362 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb"] Jul 15 13:00:05.715923 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.715887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" event={"ID":"def0c5a1-d7a2-4910-9480-50d49bdc0774","Type":"ContainerStarted","Data":"6aa97616d26471d4c0357b6d02bcacab1fcfd911d1c1b24dbec41c44f2b9e6be"} Jul 15 13:00:05.723994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.723963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tmp-dir\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.723994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.723997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfckj\" (UniqueName: \"kubernetes.io/projected/7ff538ec-5847-46ce-8a6d-f20be15ea3df-kube-api-access-kfckj\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.724194 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.724102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tls-certs\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.724194 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.724156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-model-cache\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.724262 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.724196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-dshm\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.724262 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.724240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-home\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.824853 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.824814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tmp-dir\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.824853 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.824860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfckj\" (UniqueName: \"kubernetes.io/projected/7ff538ec-5847-46ce-8a6d-f20be15ea3df-kube-api-access-kfckj\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.824930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tls-certs\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.824993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-model-cache\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.825037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-dshm\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825290 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.825128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-home\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825366 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.825340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-model-cache\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825422 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.825340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tmp-dir\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.825683 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.825660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-home\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.827750 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.827725 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-dshm\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.827900 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.827882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tls-certs\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.847322 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.847272 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfckj\" (UniqueName: \"kubernetes.io/projected/7ff538ec-5847-46ce-8a6d-f20be15ea3df-kube-api-access-kfckj\") pod \"test-model-1fe3a4-kserve-6cc4598bd5-955xb\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:05.964079 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:05.963966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:06.217868 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:06.217840 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb"] Jul 15 13:00:06.220085 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:00:06.220054 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff538ec_5847_46ce_8a6d_f20be15ea3df.slice/crio-de04b9d52422587b04952d116ddef4850bd34cccf121b91e62bdf9d97d70a002 WatchSource:0}: Error finding container de04b9d52422587b04952d116ddef4850bd34cccf121b91e62bdf9d97d70a002: Status 404 returned error can't find the container with id de04b9d52422587b04952d116ddef4850bd34cccf121b91e62bdf9d97d70a002 Jul 15 13:00:06.434336 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:06.434228 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 13:00:06.435095 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:06.434972 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 13:00:06.435335 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:06.435075 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 13:00:06.723129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:06.723086 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" event={"ID":"7ff538ec-5847-46ce-8a6d-f20be15ea3df","Type":"ContainerStarted","Data":"ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503"} Jul 15 13:00:06.723129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:06.723138 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" event={"ID":"7ff538ec-5847-46ce-8a6d-f20be15ea3df","Type":"ContainerStarted","Data":"de04b9d52422587b04952d116ddef4850bd34cccf121b91e62bdf9d97d70a002"} Jul 15 13:00:07.434785 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:07.434291 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 13:00:07.434785 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:07.434357 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 13:00:08.434887 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:08.434779 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 13:00:08.434887 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:08.434849 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 13:00:09.434420 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:09.434380 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 13:00:09.434742 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:09.434709 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 13:00:10.438524 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:10.438491 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 13:00:10.438963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:10.438792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 13:00:10.439546 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:10.439528 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 13:00:10.474739 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:10.474682 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" podStartSLOduration=5.474660138 podStartE2EDuration="5.474660138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:00:06.759346594 +0000 UTC m=+2165.972093106" watchObservedRunningTime="2026-07-15 13:00:10.474660138 +0000 UTC m=+2169.687406649" Jul 15 13:00:11.669164 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.669126 2582 patch_prober.go:28] interesting pod/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 13:00:11.669634 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.669181 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 13:00:11.762017 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.761983 2582 generic.go:356] "Generic (PLEG): container finished" podID="bba612a6-9383-4c57-a11d-e77320fe8336" containerID="6b41b6bc4856c5522f74127b98964b3b37f72210b575790b62046a14a805fc5a" exitCode=0 Jul 15 13:00:11.762184 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.762063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" event={"ID":"bba612a6-9383-4c57-a11d-e77320fe8336","Type":"ContainerDied","Data":"6b41b6bc4856c5522f74127b98964b3b37f72210b575790b62046a14a805fc5a"} Jul 15 13:00:11.762615 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.762590 2582 scope.go:117] "RemoveContainer" containerID="6b41b6bc4856c5522f74127b98964b3b37f72210b575790b62046a14a805fc5a" Jul 15 13:00:11.763642 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.763613 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:00:11.763752 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.763695 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:00:11.763801 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:11.763748 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:00:12.769901 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:12.769852 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" event={"ID":"bba612a6-9383-4c57-a11d-e77320fe8336","Type":"ContainerStarted","Data":"95ce52c876a47bacf503bad4e3d1da79c42d2f2c367580aa065141ad3d8931e4"} Jul 15 13:00:13.127073 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:13.126976 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:00:13.127221 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:13.127159 2582 patch_prober.go:28] interesting pod/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 13:00:13.127221 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:13.127205 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 13:00:14.127134 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:14.127092 2582 patch_prober.go:28] interesting pod/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 13:00:14.127538 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:14.127146 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 13:00:15.127864 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:15.127826 2582 patch_prober.go:28] interesting pod/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 13:00:15.128268 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:15.127889 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 13:00:15.964892 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:15.964848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:15.965177 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:15.964918 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:15.977489 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:15.977462 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:16.127166 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:16.127132 2582 patch_prober.go:28] interesting pod/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 13:00:16.127396 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:16.127187 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 13:00:16.800794 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:16.800765 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:00:17.126921 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:17.126820 2582 patch_prober.go:28] interesting pod/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 13:00:17.126921 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:17.126879 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 13:00:18.130830 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:18.130797 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:00:18.131224 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:18.131071 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:00:18.131900 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:18.131881 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:00:45.924328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:00:45.924293 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb"] Jul 15 13:01:00.944750 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:00.944700 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" podUID="7ff538ec-5847-46ce-8a6d-f20be15ea3df" containerName="main" containerID="cri-o://ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503" gracePeriod=45 Jul 15 13:01:01.217048 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.217022 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:01:01.295623 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfckj\" (UniqueName: \"kubernetes.io/projected/7ff538ec-5847-46ce-8a6d-f20be15ea3df-kube-api-access-kfckj\") pod \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " Jul 15 13:01:01.295821 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295654 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tmp-dir\") pod \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " Jul 15 13:01:01.295821 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295706 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-model-cache\") pod \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " Jul 15 13:01:01.295821 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295731 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-dshm\") pod \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " Jul 15 13:01:01.295821 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295756 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-home\") pod \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " Jul 15 13:01:01.295821 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295802 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tls-certs\") pod \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\" (UID: \"7ff538ec-5847-46ce-8a6d-f20be15ea3df\") " Jul 15 13:01:01.296089 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295934 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ff538ec-5847-46ce-8a6d-f20be15ea3df" (UID: "7ff538ec-5847-46ce-8a6d-f20be15ea3df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:01.296089 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.295977 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ff538ec-5847-46ce-8a6d-f20be15ea3df" (UID: "7ff538ec-5847-46ce-8a6d-f20be15ea3df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:01.296188 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.296118 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-home" (OuterVolumeSpecName: "home") pod "7ff538ec-5847-46ce-8a6d-f20be15ea3df" (UID: "7ff538ec-5847-46ce-8a6d-f20be15ea3df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:01.296241 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.296187 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-model-cache\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:01.296241 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.296206 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tmp-dir\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:01.297905 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.297886 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-dshm" (OuterVolumeSpecName: "dshm") pod "7ff538ec-5847-46ce-8a6d-f20be15ea3df" (UID: "7ff538ec-5847-46ce-8a6d-f20be15ea3df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:01.297974 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.297919 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ff538ec-5847-46ce-8a6d-f20be15ea3df" (UID: "7ff538ec-5847-46ce-8a6d-f20be15ea3df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:01:01.298026 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.298006 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff538ec-5847-46ce-8a6d-f20be15ea3df-kube-api-access-kfckj" (OuterVolumeSpecName: "kube-api-access-kfckj") pod "7ff538ec-5847-46ce-8a6d-f20be15ea3df" (UID: "7ff538ec-5847-46ce-8a6d-f20be15ea3df"). InnerVolumeSpecName "kube-api-access-kfckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:01:01.397620 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.397595 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-dshm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:01.397620 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.397616 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff538ec-5847-46ce-8a6d-f20be15ea3df-home\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:01.397769 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.397626 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff538ec-5847-46ce-8a6d-f20be15ea3df-tls-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:01.397769 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:01.397636 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfckj\" (UniqueName: \"kubernetes.io/projected/7ff538ec-5847-46ce-8a6d-f20be15ea3df-kube-api-access-kfckj\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:02.013193 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.013157 2582 generic.go:356] "Generic (PLEG): container finished" podID="7ff538ec-5847-46ce-8a6d-f20be15ea3df" containerID="ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503" exitCode=0 Jul 15 13:01:02.013694 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.013201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" event={"ID":"7ff538ec-5847-46ce-8a6d-f20be15ea3df","Type":"ContainerDied","Data":"ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503"} Jul 15 13:01:02.013694 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.013230 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" event={"ID":"7ff538ec-5847-46ce-8a6d-f20be15ea3df","Type":"ContainerDied","Data":"de04b9d52422587b04952d116ddef4850bd34cccf121b91e62bdf9d97d70a002"} Jul 15 13:01:02.013694 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.013245 2582 scope.go:117] "RemoveContainer" containerID="ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503" Jul 15 13:01:02.013694 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.013202 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb" Jul 15 13:01:02.021788 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.021766 2582 scope.go:117] "RemoveContainer" containerID="ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503" Jul 15 13:01:02.022032 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:01:02.022012 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503\": container with ID starting with ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503 not found: ID does not exist" containerID="ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503" Jul 15 13:01:02.022085 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.022039 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503"} err="failed to get container status \"ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503\": rpc error: code = NotFound desc = could not find container \"ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503\": container with ID starting with ec7fb320973ac7354e7f0f4f1e3ad1c510c9f233444821aa932529d918a04503 not found: ID does not exist" Jul 15 13:01:02.040210 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.040181 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb"] Jul 15 13:01:02.044946 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:02.044923 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1fe3a4/test-model-1fe3a4-kserve-6cc4598bd5-955xb"] Jul 15 13:01:03.315039 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:03.315003 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff538ec-5847-46ce-8a6d-f20be15ea3df" path="/var/lib/kubelet/pods/7ff538ec-5847-46ce-8a6d-f20be15ea3df/volumes" Jul 15 13:01:12.881921 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:12.881889 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m"] Jul 15 13:01:12.882317 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:12.882101 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" containerID="cri-o://6aa97616d26471d4c0357b6d02bcacab1fcfd911d1c1b24dbec41c44f2b9e6be" gracePeriod=30 Jul 15 13:01:18.094636 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.094605 2582 generic.go:356] "Generic (PLEG): container finished" podID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerID="6aa97616d26471d4c0357b6d02bcacab1fcfd911d1c1b24dbec41c44f2b9e6be" exitCode=0 Jul 15 13:01:18.095078 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.094676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" event={"ID":"def0c5a1-d7a2-4910-9480-50d49bdc0774","Type":"ContainerDied","Data":"6aa97616d26471d4c0357b6d02bcacab1fcfd911d1c1b24dbec41c44f2b9e6be"} Jul 15 13:01:18.095078 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.094740 2582 scope.go:117] "RemoveContainer" containerID="cc474a5c0b1d7a9503f13cac17dd88dd4fd95d67f9f914dd9bb62cfae0f520cb" Jul 15 13:01:18.136234 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.136206 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 13:01:18.260790 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-token\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.260949 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260801 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-data\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.260949 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260827 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-envoy\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.260949 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260843 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-certs\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.260949 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260868 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def0c5a1-d7a2-4910-9480-50d49bdc0774-istiod-ca-cert\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.260949 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-credential-socket\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.261216 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.260973 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkdnh\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-kube-api-access-lkdnh\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.261216 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261013 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-podinfo\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.261216 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261036 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-socket\") pod \"def0c5a1-d7a2-4910-9480-50d49bdc0774\" (UID: \"def0c5a1-d7a2-4910-9480-50d49bdc0774\") " Jul 15 13:01:18.261372 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261346 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:18.261435 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261380 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:18.261682 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261651 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def0c5a1-d7a2-4910-9480-50d49bdc0774-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:01:18.261682 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261674 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-data" (OuterVolumeSpecName: "istio-data") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:18.261933 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.261912 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:18.263174 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.263094 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:18.263272 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.263203 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:01:18.263458 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.263435 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-token" (OuterVolumeSpecName: "istio-token") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:01:18.263557 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.263464 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-kube-api-access-lkdnh" (OuterVolumeSpecName: "kube-api-access-lkdnh") pod "def0c5a1-d7a2-4910-9480-50d49bdc0774" (UID: "def0c5a1-d7a2-4910-9480-50d49bdc0774"). InnerVolumeSpecName "kube-api-access-lkdnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:01:18.362055 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362031 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362055 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362052 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdnh\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-kube-api-access-lkdnh\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362061 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362070 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362079 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362087 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362094 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362101 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def0c5a1-d7a2-4910-9480-50d49bdc0774-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:18.362178 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:18.362108 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def0c5a1-d7a2-4910-9480-50d49bdc0774-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:19.100555 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:19.100508 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" event={"ID":"def0c5a1-d7a2-4910-9480-50d49bdc0774","Type":"ContainerDied","Data":"74ac8ad568264677e90b34fe014f1fee085b5371586c4af261645efe0912b554"} Jul 15 13:01:19.100945 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:19.100590 2582 scope.go:117] "RemoveContainer" containerID="6aa97616d26471d4c0357b6d02bcacab1fcfd911d1c1b24dbec41c44f2b9e6be" Jul 15 13:01:19.100945 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:19.100528 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m" Jul 15 13:01:19.128370 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:19.128346 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m"] Jul 15 13:01:19.130120 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:19.130101 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe3a4-openshift-default-d65bb7865-jj77m"] Jul 15 13:01:19.315163 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:19.315131 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" path="/var/lib/kubelet/pods/def0c5a1-d7a2-4910-9480-50d49bdc0774/volumes" Jul 15 13:01:39.104926 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:39.104850 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5"] Jul 15 13:01:39.105297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:39.105118 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" containerID="cri-o://95ce52c876a47bacf503bad4e3d1da79c42d2f2c367580aa065141ad3d8931e4" gracePeriod=30 Jul 15 13:01:40.008763 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.008725 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv"] Jul 15 13:01:40.009188 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009174 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff538ec-5847-46ce-8a6d-f20be15ea3df" containerName="main" Jul 15 13:01:40.009237 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009190 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff538ec-5847-46ce-8a6d-f20be15ea3df" containerName="main" Jul 15 13:01:40.009237 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009202 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" Jul 15 13:01:40.009237 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009208 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" Jul 15 13:01:40.009237 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009220 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" Jul 15 13:01:40.009237 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009226 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" Jul 15 13:01:40.009393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009293 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" Jul 15 13:01:40.009393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009301 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="def0c5a1-d7a2-4910-9480-50d49bdc0774" containerName="istio-proxy" Jul 15 13:01:40.009393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.009308 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff538ec-5847-46ce-8a6d-f20be15ea3df" containerName="main" Jul 15 13:01:40.012239 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.012221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.015360 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.015341 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9c73fd-openshift-default-dockercfg-8hvv5\"" Jul 15 13:01:40.025480 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.025459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv"] Jul 15 13:01:40.166586 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-data\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6d4\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-kube-api-access-2s6d4\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-envoy\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166721 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istiod-ca-cert\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-certs\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-token\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-socket\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-credential-socket\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.166966 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.166912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-podinfo\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268082 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-podinfo\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268082 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-data\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268252 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6d4\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-kube-api-access-2s6d4\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268252 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-envoy\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268435 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istiod-ca-cert\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268435 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-certs\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268435 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-token\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268435 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-data\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268668 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-socket\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268668 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-credential-socket\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268888 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-socket\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268958 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-certs\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.268958 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268925 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-credential-socket\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.269034 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.268965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istiod-ca-cert\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.270450 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.270427 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-envoy\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.270572 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.270520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-podinfo\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.276895 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.276868 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-token\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.277118 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.277101 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6d4\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-kube-api-access-2s6d4\") pod \"e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.322617 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.322588 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:40.466204 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.466174 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv"] Jul 15 13:01:40.467952 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:01:40.467927 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e124de_a8f4_4e09_8368_a33885cd3bfb.slice/crio-0ab7ff40c59588776cfbe83898e7e6c1565e4dd089bb4874e419dfc7188ab8d9 WatchSource:0}: Error finding container 0ab7ff40c59588776cfbe83898e7e6c1565e4dd089bb4874e419dfc7188ab8d9: Status 404 returned error can't find the container with id 0ab7ff40c59588776cfbe83898e7e6c1565e4dd089bb4874e419dfc7188ab8d9 Jul 15 13:01:40.470195 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.470156 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:01:40.470277 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.470223 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:01:40.470277 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:40.470251 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:01:41.211036 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:41.211004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" event={"ID":"a9e124de-a8f4-4e09-8368-a33885cd3bfb","Type":"ContainerStarted","Data":"5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66"} Jul 15 13:01:41.211428 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:41.211054 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" event={"ID":"a9e124de-a8f4-4e09-8368-a33885cd3bfb","Type":"ContainerStarted","Data":"0ab7ff40c59588776cfbe83898e7e6c1565e4dd089bb4874e419dfc7188ab8d9"} Jul 15 13:01:41.238142 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:41.238096 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podStartSLOduration=2.238079946 podStartE2EDuration="2.238079946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:01:41.23637509 +0000 UTC m=+2260.449121612" watchObservedRunningTime="2026-07-15 13:01:41.238079946 +0000 UTC m=+2260.450826456" Jul 15 13:01:41.323614 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:41.323584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:41.328501 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:41.328476 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:42.216676 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:42.216638 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:42.217624 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:42.217599 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:01:44.229747 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.229720 2582 generic.go:356] "Generic (PLEG): container finished" podID="bba612a6-9383-4c57-a11d-e77320fe8336" containerID="95ce52c876a47bacf503bad4e3d1da79c42d2f2c367580aa065141ad3d8931e4" exitCode=0 Jul 15 13:01:44.230071 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.229791 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" event={"ID":"bba612a6-9383-4c57-a11d-e77320fe8336","Type":"ContainerDied","Data":"95ce52c876a47bacf503bad4e3d1da79c42d2f2c367580aa065141ad3d8931e4"} Jul 15 13:01:44.230071 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.229847 2582 scope.go:117] "RemoveContainer" containerID="6b41b6bc4856c5522f74127b98964b3b37f72210b575790b62046a14a805fc5a" Jul 15 13:01:44.362518 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.362486 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:01:44.513450 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513428 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-credential-socket\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513611 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513466 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-socket\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513611 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513489 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-data\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513611 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513520 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba612a6-9383-4c57-a11d-e77320fe8336-istio-podinfo\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513611 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513537 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-certs\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513611 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513608 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-envoy\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513874 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513679 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-istio-token\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513874 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513711 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zbsc\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-kube-api-access-6zbsc\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513874 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba612a6-9383-4c57-a11d-e77320fe8336-istiod-ca-cert\") pod \"bba612a6-9383-4c57-a11d-e77320fe8336\" (UID: \"bba612a6-9383-4c57-a11d-e77320fe8336\") " Jul 15 13:01:44.513874 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513733 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:44.513874 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513754 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:44.514124 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513954 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.514124 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.513971 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.514225 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.514166 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:44.514225 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.514185 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba612a6-9383-4c57-a11d-e77320fe8336-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:01:44.514520 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.514496 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-data" (OuterVolumeSpecName: "istio-data") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:44.515836 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.515815 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:01:44.515996 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.515972 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-kube-api-access-6zbsc" (OuterVolumeSpecName: "kube-api-access-6zbsc") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "kube-api-access-6zbsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:01:44.516278 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.516254 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba612a6-9383-4c57-a11d-e77320fe8336-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:01:44.516378 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.516273 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-istio-token" (OuterVolumeSpecName: "istio-token") pod "bba612a6-9383-4c57-a11d-e77320fe8336" (UID: "bba612a6-9383-4c57-a11d-e77320fe8336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:01:44.615145 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615115 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.615145 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615143 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba612a6-9383-4c57-a11d-e77320fe8336-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.615145 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615154 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.615287 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615162 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba612a6-9383-4c57-a11d-e77320fe8336-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.615287 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615170 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.615287 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615177 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbsc\" (UniqueName: \"kubernetes.io/projected/bba612a6-9383-4c57-a11d-e77320fe8336-kube-api-access-6zbsc\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:44.615287 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:44.615185 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba612a6-9383-4c57-a11d-e77320fe8336-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:01:45.236478 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:45.236452 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" Jul 15 13:01:45.236478 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:45.236456 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5" event={"ID":"bba612a6-9383-4c57-a11d-e77320fe8336","Type":"ContainerDied","Data":"49fe7b838b5bc740e50c1141596fbd6a8cc9d88e148f6eb14d18af2ec9ab2815"} Jul 15 13:01:45.236972 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:45.236518 2582 scope.go:117] "RemoveContainer" containerID="95ce52c876a47bacf503bad4e3d1da79c42d2f2c367580aa065141ad3d8931e4" Jul 15 13:01:45.262526 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:45.262503 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5"] Jul 15 13:01:45.265550 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:45.265529 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-34f439-openshift-default-6454f74cc7-mwbv5"] Jul 15 13:01:45.315064 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:45.315040 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" path="/var/lib/kubelet/pods/bba612a6-9383-4c57-a11d-e77320fe8336/volumes" Jul 15 13:01:48.613022 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.612988 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l"] Jul 15 13:01:48.613413 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.613400 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" Jul 15 13:01:48.613413 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.613414 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" Jul 15 13:01:48.613793 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.613422 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" Jul 15 13:01:48.613793 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.613427 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" Jul 15 13:01:48.613793 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.613483 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" Jul 15 13:01:48.617923 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.617902 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.620976 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.620955 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8a1d21-openshift-default-dockercfg-lwf6k\"" Jul 15 13:01:48.647227 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.647205 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l"] Jul 15 13:01:48.752642 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-certs\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752816 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-socket\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752816 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-podinfo\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752816 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-envoy\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752816 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-data\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752959 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452ae92e-d62c-452c-bb1d-acde654c21d9-istiod-ca-cert\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752959 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz84f\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-kube-api-access-hz84f\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.752959 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-credential-socket\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.753049 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.752966 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-token\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853713 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-token\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853713 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-certs\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-socket\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-podinfo\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-envoy\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-data\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853877 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452ae92e-d62c-452c-bb1d-acde654c21d9-istiod-ca-cert\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.853963 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz84f\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-kube-api-access-hz84f\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.854237 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.853972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-credential-socket\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.854294 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.854251 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-socket\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.854379 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.854353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-certs\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.854465 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.854441 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-credential-socket\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.854539 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.854518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-data\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.854768 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.854751 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452ae92e-d62c-452c-bb1d-acde654c21d9-istiod-ca-cert\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.856251 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.856233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-envoy\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.856461 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.856442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-podinfo\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.869173 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.869113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz84f\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-kube-api-access-hz84f\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.870447 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.870422 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-token\") pod \"e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:48.927578 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:48.927529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:49.089324 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.089301 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l"] Jul 15 13:01:49.091914 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:01:49.091885 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ae92e_d62c_452c_bb1d_acde654c21d9.slice/crio-8e50133112c463358a5d172805cf4163042b26ee146d7ebd4964e88256740641 WatchSource:0}: Error finding container 8e50133112c463358a5d172805cf4163042b26ee146d7ebd4964e88256740641: Status 404 returned error can't find the container with id 8e50133112c463358a5d172805cf4163042b26ee146d7ebd4964e88256740641 Jul 15 13:01:49.093998 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.093967 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:01:49.094081 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.094029 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:01:49.094081 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.094055 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:01:49.260034 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.259995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" event={"ID":"452ae92e-d62c-452c-bb1d-acde654c21d9","Type":"ContainerStarted","Data":"412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1"} Jul 15 13:01:49.260192 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.260056 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" event={"ID":"452ae92e-d62c-452c-bb1d-acde654c21d9","Type":"ContainerStarted","Data":"8e50133112c463358a5d172805cf4163042b26ee146d7ebd4964e88256740641"} Jul 15 13:01:49.928611 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:49.928574 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:50.932199 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:50.932172 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:50.959361 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:50.959313 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" podStartSLOduration=2.959299868 podStartE2EDuration="2.959299868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:01:49.300942999 +0000 UTC m=+2268.513689578" watchObservedRunningTime="2026-07-15 13:01:50.959299868 +0000 UTC m=+2270.172046379" Jul 15 13:01:51.269819 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:51.269791 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:51.270737 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:51.270715 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:01:58.258824 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:58.258700 2582 patch_prober.go:28] interesting pod/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 13:01:58.258824 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:01:58.258780 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 13:02:00.315895 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:00.315816 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerID="5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66" exitCode=0 Jul 15 13:02:00.316405 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:00.315896 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" event={"ID":"a9e124de-a8f4-4e09-8368-a33885cd3bfb","Type":"ContainerDied","Data":"5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66"} Jul 15 13:02:00.316472 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:00.316425 2582 scope.go:117] "RemoveContainer" containerID="5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66" Jul 15 13:02:00.317599 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:00.317545 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:00.317696 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:00.317656 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:00.317747 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:00.317694 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:01.322638 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:01.322600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" event={"ID":"a9e124de-a8f4-4e09-8368-a33885cd3bfb","Type":"ContainerStarted","Data":"cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29"} Jul 15 13:02:02.323180 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:02.323142 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:02:02.325260 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:02.323327 2582 patch_prober.go:28] interesting pod/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:02:02.325260 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:02.323372 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:02:03.322962 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.322928 2582 patch_prober.go:28] interesting pod/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:02:03.323137 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.322981 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:02:03.334503 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.334466 2582 generic.go:356] "Generic (PLEG): container finished" podID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerID="412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1" exitCode=0 Jul 15 13:02:03.334936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.334621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" event={"ID":"452ae92e-d62c-452c-bb1d-acde654c21d9","Type":"ContainerDied","Data":"412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1"} Jul 15 13:02:03.335145 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.335122 2582 scope.go:117] "RemoveContainer" containerID="412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1" Jul 15 13:02:03.336428 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.336392 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:03.336524 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.336483 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:03.336608 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:03.336531 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:04.323103 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:04.323071 2582 patch_prober.go:28] interesting pod/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:02:04.323282 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:04.323126 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:02:04.345543 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:04.345501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" event={"ID":"452ae92e-d62c-452c-bb1d-acde654c21d9","Type":"ContainerStarted","Data":"7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7"} Jul 15 13:02:04.927885 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:04.927854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:02:04.928949 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:04.928918 2582 patch_prober.go:28] interesting pod/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 15 13:02:04.929074 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:04.928959 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 15 13:02:05.323874 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:05.323832 2582 patch_prober.go:28] interesting pod/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:02:05.324083 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:05.323884 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:02:05.928227 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:05.928192 2582 patch_prober.go:28] interesting pod/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 15 13:02:05.928657 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:05.928242 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 15 13:02:06.269909 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:06.269877 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:02:06.326142 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:06.326120 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:02:06.326533 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:06.326512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:02:06.327217 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:06.327201 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:02:06.927891 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:06.927860 2582 patch_prober.go:28] interesting pod/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 15 13:02:06.928050 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:06.927906 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 15 13:02:07.928504 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:07.928469 2582 patch_prober.go:28] interesting pod/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 15 13:02:07.928886 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:07.928520 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 15 13:02:08.931241 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:08.931208 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:02:08.932121 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:08.932099 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:02:29.808122 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:29.808091 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv"] Jul 15 13:02:29.808906 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:29.808306 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" containerID="cri-o://cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29" gracePeriod=30 Jul 15 13:02:35.057459 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.057433 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:02:35.213798 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.213717 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-envoy\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.213798 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.213763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-socket\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.213987 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.213811 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s6d4\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-kube-api-access-2s6d4\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.213987 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.213858 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istiod-ca-cert\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.213987 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.213933 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-token\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.213987 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.213964 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-podinfo\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.214201 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214007 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-certs\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.214201 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214059 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-credential-socket\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.214201 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214078 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:02:35.214201 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214096 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-data\") pod \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\" (UID: \"a9e124de-a8f4-4e09-8368-a33885cd3bfb\") " Jul 15 13:02:35.214419 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214295 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:02:35.214419 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214347 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:02:35.214524 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214489 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.214524 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214509 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.214524 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214522 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.214683 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.214536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:02:35.215246 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.215218 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:02:35.216144 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.216100 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-kube-api-access-2s6d4" (OuterVolumeSpecName: "kube-api-access-2s6d4") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "kube-api-access-2s6d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:02:35.216292 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.216155 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:02:35.216292 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.216192 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:02:35.216405 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.216386 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9e124de-a8f4-4e09-8368-a33885cd3bfb" (UID: "a9e124de-a8f4-4e09-8368-a33885cd3bfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:02:35.315095 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.315058 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.315095 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.315088 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.315095 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.315100 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.315328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.315112 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6d4\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-kube-api-access-2s6d4\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.315328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.315124 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.315328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.315135 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e124de-a8f4-4e09-8368-a33885cd3bfb-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:02:35.501778 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.501685 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerID="cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29" exitCode=0 Jul 15 13:02:35.501778 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.501734 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" Jul 15 13:02:35.501778 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.501754 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" event={"ID":"a9e124de-a8f4-4e09-8368-a33885cd3bfb","Type":"ContainerDied","Data":"cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29"} Jul 15 13:02:35.502053 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.501798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv" event={"ID":"a9e124de-a8f4-4e09-8368-a33885cd3bfb","Type":"ContainerDied","Data":"0ab7ff40c59588776cfbe83898e7e6c1565e4dd089bb4874e419dfc7188ab8d9"} Jul 15 13:02:35.502053 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.501820 2582 scope.go:117] "RemoveContainer" containerID="cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29" Jul 15 13:02:35.510452 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.510434 2582 scope.go:117] "RemoveContainer" containerID="5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66" Jul 15 13:02:35.518684 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.518659 2582 scope.go:117] "RemoveContainer" containerID="cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29" Jul 15 13:02:35.519020 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:02:35.518991 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29\": container with ID starting with cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29 not found: ID does not exist" containerID="cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29" Jul 15 13:02:35.519111 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.519030 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29"} err="failed to get container status \"cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29\": rpc error: code = NotFound desc = could not find container \"cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29\": container with ID starting with cfaf993bb6f8731d0a95e65a2228f1da18841d2f0764221a8e879b01b9b47d29 not found: ID does not exist" Jul 15 13:02:35.519111 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.519054 2582 scope.go:117] "RemoveContainer" containerID="5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66" Jul 15 13:02:35.519498 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:02:35.519470 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66\": container with ID starting with 5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66 not found: ID does not exist" containerID="5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66" Jul 15 13:02:35.519928 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.519515 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66"} err="failed to get container status \"5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66\": rpc error: code = NotFound desc = could not find container \"5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66\": container with ID starting with 5a3eb35b7ec3a509a7faf2bf0e27c5097f450d08a767ec108ae67d1b5887cf66 not found: ID does not exist" Jul 15 13:02:35.524475 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.524452 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv"] Jul 15 13:02:35.527761 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:35.527734 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9c73fd-openshift-default-7c99d554bc-dfvmv"] Jul 15 13:02:37.314923 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:37.314893 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" path="/var/lib/kubelet/pods/a9e124de-a8f4-4e09-8368-a33885cd3bfb/volumes" Jul 15 13:02:56.099129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:56.099048 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l"] Jul 15 13:02:56.099513 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:56.099252 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" containerID="cri-o://7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7" gracePeriod=30 Jul 15 13:02:57.054328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.054294 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn"] Jul 15 13:02:57.054730 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.054715 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" Jul 15 13:02:57.054791 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.054734 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" Jul 15 13:02:57.054832 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.054814 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" Jul 15 13:02:57.054832 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.054824 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" Jul 15 13:02:57.054894 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.054834 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba612a6-9383-4c57-a11d-e77320fe8336" containerName="istio-proxy" Jul 15 13:02:57.056813 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.056797 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.060470 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.060450 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ce0265-openshift-default-dockercfg-xz7f4\"" Jul 15 13:02:57.102173 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istiod-ca-cert\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102173 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5w8m\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-kube-api-access-p5w8m\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-token\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102311 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-envoy\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-data\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-socket\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-certs\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-credential-socket\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.102591 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.102449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-podinfo\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.117798 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.117771 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn"] Jul 15 13:02:57.203578 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5w8m\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-kube-api-access-p5w8m\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.203720 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-token\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.203720 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-envoy\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.203720 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-data\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.203878 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-socket\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.203878 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-certs\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204001 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-credential-socket\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204001 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-podinfo\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204001 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.203937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-data\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204164 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.204002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istiod-ca-cert\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204217 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.204154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-socket\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204217 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.204181 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-certs\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.204225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-credential-socket\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.204586 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.204544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istiod-ca-cert\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.205934 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.205912 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-envoy\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.206174 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.206154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-podinfo\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.218946 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.218926 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-token\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.219239 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.219216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5w8m\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-kube-api-access-p5w8m\") pod \"e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.367333 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.367245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:57.504765 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.504738 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn"] Jul 15 13:02:57.506415 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:02:57.506389 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165bc244_8c8b_4343_89e8_cc0bb4d1e0bd.slice/crio-e65894f202876c316f6cf3d787f9e08edbeb705184a19e12907f1a854fd33343 WatchSource:0}: Error finding container e65894f202876c316f6cf3d787f9e08edbeb705184a19e12907f1a854fd33343: Status 404 returned error can't find the container with id e65894f202876c316f6cf3d787f9e08edbeb705184a19e12907f1a854fd33343 Jul 15 13:02:57.508461 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.508430 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:57.508596 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.508505 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:57.508596 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.508549 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:02:57.611496 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.611458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" event={"ID":"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd","Type":"ContainerStarted","Data":"0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444"} Jul 15 13:02:57.611658 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.611529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" event={"ID":"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd","Type":"ContainerStarted","Data":"e65894f202876c316f6cf3d787f9e08edbeb705184a19e12907f1a854fd33343"} Jul 15 13:02:57.656170 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:57.656063 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podStartSLOduration=0.655988227 podStartE2EDuration="655.988227ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:02:57.655188511 +0000 UTC m=+2336.867935034" watchObservedRunningTime="2026-07-15 13:02:57.655988227 +0000 UTC m=+2336.868734738" Jul 15 13:02:58.368076 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:58.368039 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:58.372798 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:58.372773 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:58.616448 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:58.616409 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:02:58.617466 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:02:58.617440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:03:01.349990 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.349965 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:03:01.443356 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-envoy\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443356 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443341 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz84f\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-kube-api-access-hz84f\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443579 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443367 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-socket\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443579 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-certs\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443579 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443429 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-podinfo\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443579 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443487 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-credential-socket\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443579 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443527 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452ae92e-d62c-452c-bb1d-acde654c21d9-istiod-ca-cert\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443579 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443550 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-token\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443931 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-data\") pod \"452ae92e-d62c-452c-bb1d-acde654c21d9\" (UID: \"452ae92e-d62c-452c-bb1d-acde654c21d9\") " Jul 15 13:03:01.443931 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443758 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:03:01.443931 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443776 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:03:01.444075 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.443983 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452ae92e-d62c-452c-bb1d-acde654c21d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:03:01.444075 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.444037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:03:01.444181 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.444078 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.444181 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.444100 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.444181 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.444115 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452ae92e-d62c-452c-bb1d-acde654c21d9-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.444417 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.444401 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:03:01.445836 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.445809 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:03:01.445836 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.445831 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:03:01.446010 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.445995 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-kube-api-access-hz84f" (OuterVolumeSpecName: "kube-api-access-hz84f") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "kube-api-access-hz84f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:03:01.446049 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.446015 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "452ae92e-d62c-452c-bb1d-acde654c21d9" (UID: "452ae92e-d62c-452c-bb1d-acde654c21d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:03:01.544939 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.544916 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.544939 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.544936 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.545080 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.544945 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.545080 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.544954 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz84f\" (UniqueName: \"kubernetes.io/projected/452ae92e-d62c-452c-bb1d-acde654c21d9-kube-api-access-hz84f\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.545080 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.544962 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452ae92e-d62c-452c-bb1d-acde654c21d9-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.545080 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.544971 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452ae92e-d62c-452c-bb1d-acde654c21d9-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:03:01.632207 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.632186 2582 generic.go:356] "Generic (PLEG): container finished" podID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerID="7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7" exitCode=0 Jul 15 13:03:01.632322 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.632232 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" Jul 15 13:03:01.632322 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.632274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" event={"ID":"452ae92e-d62c-452c-bb1d-acde654c21d9","Type":"ContainerDied","Data":"7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7"} Jul 15 13:03:01.632399 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.632329 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l" event={"ID":"452ae92e-d62c-452c-bb1d-acde654c21d9","Type":"ContainerDied","Data":"8e50133112c463358a5d172805cf4163042b26ee146d7ebd4964e88256740641"} Jul 15 13:03:01.632399 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.632353 2582 scope.go:117] "RemoveContainer" containerID="7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7" Jul 15 13:03:01.641076 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.640967 2582 scope.go:117] "RemoveContainer" containerID="412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1" Jul 15 13:03:01.648623 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.648609 2582 scope.go:117] "RemoveContainer" containerID="7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7" Jul 15 13:03:01.648883 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:03:01.648864 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7\": container with ID starting with 7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7 not found: ID does not exist" containerID="7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7" Jul 15 13:03:01.648935 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.648891 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7"} err="failed to get container status \"7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7\": rpc error: code = NotFound desc = could not find container \"7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7\": container with ID starting with 7b8b530f77eae5e78ce55089c59076b0d6ea4996ce27b62b0b0228fef872c5b7 not found: ID does not exist" Jul 15 13:03:01.648935 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.648908 2582 scope.go:117] "RemoveContainer" containerID="412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1" Jul 15 13:03:01.649115 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:03:01.649098 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1\": container with ID starting with 412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1 not found: ID does not exist" containerID="412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1" Jul 15 13:03:01.649155 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.649121 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1"} err="failed to get container status \"412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1\": rpc error: code = NotFound desc = could not find container \"412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1\": container with ID starting with 412b080f8434b6b1479ab4ad9628a96f134b81c2695f89d4b234e682382d56c1 not found: ID does not exist" Jul 15 13:03:01.657538 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.657517 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l"] Jul 15 13:03:01.661602 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:01.661582 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8a1d21-openshift-default-76c47b89c5-tgd8l"] Jul 15 13:03:03.320742 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:03.320711 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" path="/var/lib/kubelet/pods/452ae92e-d62c-452c-bb1d-acde654c21d9/volumes" Jul 15 13:03:04.836092 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836056 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6"] Jul 15 13:03:04.836476 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836462 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" Jul 15 13:03:04.836476 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836477 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" Jul 15 13:03:04.836612 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836487 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" Jul 15 13:03:04.836612 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836493 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" Jul 15 13:03:04.836612 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836506 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" Jul 15 13:03:04.836612 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836511 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e124de-a8f4-4e09-8368-a33885cd3bfb" containerName="istio-proxy" Jul 15 13:03:04.836612 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836593 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" Jul 15 13:03:04.836612 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.836601 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="452ae92e-d62c-452c-bb1d-acde654c21d9" containerName="istio-proxy" Jul 15 13:03:04.840007 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.839988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.843159 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.843140 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-958618-openshift-default-dockercfg-m76zn\"" Jul 15 13:03:04.890383 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.890357 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6"] Jul 15 13:03:04.978043 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-socket\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978228 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-data\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978228 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-credential-socket\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978228 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978188 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-envoy\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978384 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmhm\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-kube-api-access-2nmhm\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978384 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-certs\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978384 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978292 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istiod-ca-cert\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978499 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-podinfo\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:04.978499 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:04.978406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-token\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.078937 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.078905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-data\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079132 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.078957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-credential-socket\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079132 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-envoy\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079132 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmhm\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-kube-api-access-2nmhm\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079132 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079073 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-certs\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079132 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istiod-ca-cert\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-podinfo\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-token\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-socket\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079295 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-data\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-credential-socket\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079743 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-socket\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079743 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079653 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-certs\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.079857 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.079833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istiod-ca-cert\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.081162 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.081136 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-envoy\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.081557 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.081537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-podinfo\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.088086 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.088031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-token\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.088301 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.088283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmhm\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-kube-api-access-2nmhm\") pod \"e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.150653 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.150627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:05.294513 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.294484 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6"] Jul 15 13:03:05.295543 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:03:05.295514 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9bf051_e5fc_4da7_9872_d1d2550087bb.slice/crio-aaf63605e5b991ea8b1922f0469ba58bf0d1f4db8c79e2323aa7dfadbdb828b0 WatchSource:0}: Error finding container aaf63605e5b991ea8b1922f0469ba58bf0d1f4db8c79e2323aa7dfadbdb828b0: Status 404 returned error can't find the container with id aaf63605e5b991ea8b1922f0469ba58bf0d1f4db8c79e2323aa7dfadbdb828b0 Jul 15 13:03:05.297820 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.297787 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:05.297927 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.297863 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:05.297927 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.297890 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:05.654196 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.654113 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" event={"ID":"2d9bf051-e5fc-4da7-9872-d1d2550087bb","Type":"ContainerStarted","Data":"0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc"} Jul 15 13:03:05.654196 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.654163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" event={"ID":"2d9bf051-e5fc-4da7-9872-d1d2550087bb","Type":"ContainerStarted","Data":"aaf63605e5b991ea8b1922f0469ba58bf0d1f4db8c79e2323aa7dfadbdb828b0"} Jul 15 13:03:05.680019 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:05.679971 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" podStartSLOduration=1.679954883 podStartE2EDuration="1.679954883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:03:05.678774995 +0000 UTC m=+2344.891521518" watchObservedRunningTime="2026-07-15 13:03:05.679954883 +0000 UTC m=+2344.892701394" Jul 15 13:03:06.151695 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:06.151658 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:06.156667 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:06.156642 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:06.660440 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:06.660408 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:06.661529 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:06.661509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:13.617818 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.617788 2582 patch_prober.go:28] interesting pod/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 13:03:13.618201 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.617849 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 13:03:13.697691 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.697612 2582 generic.go:356] "Generic (PLEG): container finished" podID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerID="0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444" exitCode=0 Jul 15 13:03:13.697691 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.697649 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" event={"ID":"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd","Type":"ContainerDied","Data":"0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444"} Jul 15 13:03:13.698030 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.698017 2582 scope.go:117] "RemoveContainer" containerID="0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444" Jul 15 13:03:13.699022 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.698993 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:13.699092 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.699065 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:13.699134 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:13.699102 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:14.704382 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:14.704348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" event={"ID":"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd","Type":"ContainerStarted","Data":"c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803"} Jul 15 13:03:15.368368 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:15.367899 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:03:15.368368 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:15.368033 2582 patch_prober.go:28] interesting pod/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 13:03:15.368368 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:15.368081 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 13:03:16.368188 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:16.368151 2582 patch_prober.go:28] interesting pod/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 13:03:16.368660 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:16.368223 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 13:03:17.368091 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:17.368049 2582 patch_prober.go:28] interesting pod/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 13:03:17.368294 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:17.368112 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 13:03:18.368615 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:18.368536 2582 patch_prober.go:28] interesting pod/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 13:03:18.368993 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:18.368648 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 13:03:19.372679 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.372645 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:03:19.373146 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.372897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:03:19.373639 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.373619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:03:19.729653 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.729545 2582 generic.go:356] "Generic (PLEG): container finished" podID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerID="0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc" exitCode=0 Jul 15 13:03:19.729653 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.729617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" event={"ID":"2d9bf051-e5fc-4da7-9872-d1d2550087bb","Type":"ContainerDied","Data":"0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc"} Jul 15 13:03:19.730171 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.730151 2582 scope.go:117] "RemoveContainer" containerID="0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc" Jul 15 13:03:19.731162 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.731137 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:19.731252 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.731217 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:19.731252 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:19.731245 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:03:20.741661 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:20.741617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" event={"ID":"2d9bf051-e5fc-4da7-9872-d1d2550087bb","Type":"ContainerStarted","Data":"3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116"} Jul 15 13:03:21.151576 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:21.151518 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:21.152777 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:21.152746 2582 patch_prober.go:28] interesting pod/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:03:21.152909 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:21.152805 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:03:21.660615 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:21.660584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:22.152034 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:22.151991 2582 patch_prober.go:28] interesting pod/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:03:22.152531 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:22.152057 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:03:23.151814 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:23.151772 2582 patch_prober.go:28] interesting pod/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:03:23.152030 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:23.151844 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:03:24.151297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:24.151259 2582 patch_prober.go:28] interesting pod/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:03:24.151930 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:24.151893 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:03:25.155282 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:25.155254 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:25.156212 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:25.156193 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:03:28.760464 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.760432 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp"] Jul 15 13:03:28.762869 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.762847 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.766472 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.766445 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"kube-root-ca.crt\"" Jul 15 13:03:28.767974 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.767952 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"auth-test-model-ce0265-kserve-self-signed-certs\"" Jul 15 13:03:28.767974 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.767957 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"openshift-service-ca.crt\"" Jul 15 13:03:28.768240 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.768227 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"default-dockercfg-lbnzm\"" Jul 15 13:03:28.799007 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.798980 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp"] Jul 15 13:03:28.835261 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.835234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-model-cache\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.835387 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.835270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ca8efc-da6c-49db-bf59-0544af273952-tls-certs\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.835387 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.835360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-home\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.835509 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.835426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqwz\" (UniqueName: \"kubernetes.io/projected/50ca8efc-da6c-49db-bf59-0544af273952-kube-api-access-5pqwz\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.835509 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.835474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-dshm\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.835509 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.835503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-tmp-dir\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.936811 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.936783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-home\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.936936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.936826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqwz\" (UniqueName: \"kubernetes.io/projected/50ca8efc-da6c-49db-bf59-0544af273952-kube-api-access-5pqwz\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.936936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.936852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-dshm\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.936936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.936870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-tmp-dir\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.936936 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.936919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-model-cache\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.937146 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.936955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ca8efc-da6c-49db-bf59-0544af273952-tls-certs\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.937205 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.937183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-home\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.937353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.937333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-tmp-dir\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.937418 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.937371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-model-cache\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.939102 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.939087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-dshm\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.939205 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.939189 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ca8efc-da6c-49db-bf59-0544af273952-tls-certs\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:28.947472 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:28.947447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqwz\" (UniqueName: \"kubernetes.io/projected/50ca8efc-da6c-49db-bf59-0544af273952-kube-api-access-5pqwz\") pod \"auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:29.072627 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:29.072539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:29.210066 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:29.210043 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp"] Jul 15 13:03:29.212341 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:03:29.212308 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50ca8efc_da6c_49db_bf59_0544af273952.slice/crio-3dd9ce33327fab3fdaae9f66c89dde86c1992ffd18ef92a2de14cb59470cc16a WatchSource:0}: Error finding container 3dd9ce33327fab3fdaae9f66c89dde86c1992ffd18ef92a2de14cb59470cc16a: Status 404 returned error can't find the container with id 3dd9ce33327fab3fdaae9f66c89dde86c1992ffd18ef92a2de14cb59470cc16a Jul 15 13:03:29.794139 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:29.794104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" event={"ID":"50ca8efc-da6c-49db-bf59-0544af273952","Type":"ContainerStarted","Data":"073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f"} Jul 15 13:03:29.794139 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:29.794142 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" event={"ID":"50ca8efc-da6c-49db-bf59-0544af273952","Type":"ContainerStarted","Data":"3dd9ce33327fab3fdaae9f66c89dde86c1992ffd18ef92a2de14cb59470cc16a"} Jul 15 13:03:29.838145 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:29.838099 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" podStartSLOduration=1.838083921 podStartE2EDuration="1.838083921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:03:29.835936573 +0000 UTC m=+2369.048683083" watchObservedRunningTime="2026-07-15 13:03:29.838083921 +0000 UTC m=+2369.050830431" Jul 15 13:03:39.073553 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:39.073518 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:39.074177 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:39.073608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:39.085760 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:39.085733 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:03:39.853650 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:03:39.853623 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:04:01.413478 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:04:01.413449 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 13:04:01.420754 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:04:01.420733 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 13:06:12.810015 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:12.809919 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp"] Jul 15 13:06:27.830801 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:27.830758 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" podUID="50ca8efc-da6c-49db-bf59-0544af273952" containerName="main" containerID="cri-o://073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f" gracePeriod=45 Jul 15 13:06:28.087273 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.087206 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:06:28.156682 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-model-cache\") pod \"50ca8efc-da6c-49db-bf59-0544af273952\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " Jul 15 13:06:28.156682 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156689 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ca8efc-da6c-49db-bf59-0544af273952-tls-certs\") pod \"50ca8efc-da6c-49db-bf59-0544af273952\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " Jul 15 13:06:28.156994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156728 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-dshm\") pod \"50ca8efc-da6c-49db-bf59-0544af273952\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " Jul 15 13:06:28.156994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156745 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-home\") pod \"50ca8efc-da6c-49db-bf59-0544af273952\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " Jul 15 13:06:28.156994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156788 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-tmp-dir\") pod \"50ca8efc-da6c-49db-bf59-0544af273952\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " Jul 15 13:06:28.156994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156910 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pqwz\" (UniqueName: \"kubernetes.io/projected/50ca8efc-da6c-49db-bf59-0544af273952-kube-api-access-5pqwz\") pod \"50ca8efc-da6c-49db-bf59-0544af273952\" (UID: \"50ca8efc-da6c-49db-bf59-0544af273952\") " Jul 15 13:06:28.156994 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.156944 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-model-cache" (OuterVolumeSpecName: "model-cache") pod "50ca8efc-da6c-49db-bf59-0544af273952" (UID: "50ca8efc-da6c-49db-bf59-0544af273952"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:06:28.157270 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.157058 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-home" (OuterVolumeSpecName: "home") pod "50ca8efc-da6c-49db-bf59-0544af273952" (UID: "50ca8efc-da6c-49db-bf59-0544af273952"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:06:28.157270 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.157204 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50ca8efc-da6c-49db-bf59-0544af273952" (UID: "50ca8efc-da6c-49db-bf59-0544af273952"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:06:28.157378 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.157336 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-home\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:06:28.157428 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.157382 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-tmp-dir\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:06:28.157428 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.157399 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-model-cache\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:06:28.159173 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.159152 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ca8efc-da6c-49db-bf59-0544af273952-kube-api-access-5pqwz" (OuterVolumeSpecName: "kube-api-access-5pqwz") pod "50ca8efc-da6c-49db-bf59-0544af273952" (UID: "50ca8efc-da6c-49db-bf59-0544af273952"). InnerVolumeSpecName "kube-api-access-5pqwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:06:28.159376 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.159357 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ca8efc-da6c-49db-bf59-0544af273952-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50ca8efc-da6c-49db-bf59-0544af273952" (UID: "50ca8efc-da6c-49db-bf59-0544af273952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:06:28.159431 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.159374 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-dshm" (OuterVolumeSpecName: "dshm") pod "50ca8efc-da6c-49db-bf59-0544af273952" (UID: "50ca8efc-da6c-49db-bf59-0544af273952"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:06:28.258141 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.258100 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pqwz\" (UniqueName: \"kubernetes.io/projected/50ca8efc-da6c-49db-bf59-0544af273952-kube-api-access-5pqwz\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:06:28.258141 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.258132 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ca8efc-da6c-49db-bf59-0544af273952-tls-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:06:28.258141 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.258144 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ca8efc-da6c-49db-bf59-0544af273952-dshm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:06:28.673402 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.673363 2582 generic.go:356] "Generic (PLEG): container finished" podID="50ca8efc-da6c-49db-bf59-0544af273952" containerID="073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f" exitCode=0 Jul 15 13:06:28.673646 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.673426 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" Jul 15 13:06:28.673646 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.673453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" event={"ID":"50ca8efc-da6c-49db-bf59-0544af273952","Type":"ContainerDied","Data":"073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f"} Jul 15 13:06:28.673646 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.673506 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp" event={"ID":"50ca8efc-da6c-49db-bf59-0544af273952","Type":"ContainerDied","Data":"3dd9ce33327fab3fdaae9f66c89dde86c1992ffd18ef92a2de14cb59470cc16a"} Jul 15 13:06:28.673646 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.673529 2582 scope.go:117] "RemoveContainer" containerID="073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f" Jul 15 13:06:28.683406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.683387 2582 scope.go:117] "RemoveContainer" containerID="073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f" Jul 15 13:06:28.683678 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:06:28.683658 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f\": container with ID starting with 073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f not found: ID does not exist" containerID="073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f" Jul 15 13:06:28.683752 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.683692 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f"} err="failed to get container status \"073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f\": rpc error: code = NotFound desc = could not find container \"073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f\": container with ID starting with 073a2783bed4d8aafe8800f2cbcba2ab29aecc93b136e4639fb8672d1e86a82f not found: ID does not exist" Jul 15 13:06:28.700370 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.700338 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp"] Jul 15 13:06:28.703575 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:28.703544 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/auth-test-model-ce0265-kserve-8bcf5bb75-lgfrp"] Jul 15 13:06:29.316784 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:29.316747 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ca8efc-da6c-49db-bf59-0544af273952" path="/var/lib/kubelet/pods/50ca8efc-da6c-49db-bf59-0544af273952/volumes" Jul 15 13:06:38.012733 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.012689 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg"] Jul 15 13:06:38.013152 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.013122 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ca8efc-da6c-49db-bf59-0544af273952" containerName="main" Jul 15 13:06:38.013152 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.013137 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ca8efc-da6c-49db-bf59-0544af273952" containerName="main" Jul 15 13:06:38.013228 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.013208 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ca8efc-da6c-49db-bf59-0544af273952" containerName="main" Jul 15 13:06:38.015875 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.015853 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.021689 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.021645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-958618\"/\"default-dockercfg-nlcgp\"" Jul 15 13:06:38.021859 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.021716 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-958618\"/\"kube-root-ca.crt\"" Jul 15 13:06:38.021859 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.021769 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-958618\"/\"openshift-service-ca.crt\"" Jul 15 13:06:38.021859 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.021823 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-958618\"/\"sub-test-model-958618-kserve-self-signed-certs\"" Jul 15 13:06:38.038269 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.038233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg"] Jul 15 13:06:38.162134 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.162098 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-model-cache\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.162134 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.162145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9f722-a00a-4668-aa7a-609eddd46bb4-tls-certs\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.162394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.162218 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-dshm\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.162394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.162284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c254\" (UniqueName: \"kubernetes.io/projected/aba9f722-a00a-4668-aa7a-609eddd46bb4-kube-api-access-4c254\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.162394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.162359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-home\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.162394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.162380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-tmp-dir\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263021 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.262918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-model-cache\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263021 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.262979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9f722-a00a-4668-aa7a-609eddd46bb4-tls-certs\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263021 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-dshm\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263320 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c254\" (UniqueName: \"kubernetes.io/projected/aba9f722-a00a-4668-aa7a-609eddd46bb4-kube-api-access-4c254\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263320 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-home\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263320 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-tmp-dir\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263500 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-model-cache\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263500 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263449 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-tmp-dir\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.263659 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.263496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-home\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.265461 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.265438 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-dshm\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.265557 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.265537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9f722-a00a-4668-aa7a-609eddd46bb4-tls-certs\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.276210 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.276180 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c254\" (UniqueName: \"kubernetes.io/projected/aba9f722-a00a-4668-aa7a-609eddd46bb4-kube-api-access-4c254\") pod \"sub-test-model-958618-kserve-7c58686885-7llgg\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.325980 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.325934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:38.468831 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.468717 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg"] Jul 15 13:06:38.471233 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:06:38.471157 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaba9f722_a00a_4668_aa7a_609eddd46bb4.slice/crio-cb26d990c4660a9d7d537df9cfd56c4ebe09082a4156f874257fb86c2ed896e5 WatchSource:0}: Error finding container cb26d990c4660a9d7d537df9cfd56c4ebe09082a4156f874257fb86c2ed896e5: Status 404 returned error can't find the container with id cb26d990c4660a9d7d537df9cfd56c4ebe09082a4156f874257fb86c2ed896e5 Jul 15 13:06:38.725973 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.725937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" event={"ID":"aba9f722-a00a-4668-aa7a-609eddd46bb4","Type":"ContainerStarted","Data":"8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4"} Jul 15 13:06:38.726155 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.725982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" event={"ID":"aba9f722-a00a-4668-aa7a-609eddd46bb4","Type":"ContainerStarted","Data":"cb26d990c4660a9d7d537df9cfd56c4ebe09082a4156f874257fb86c2ed896e5"} Jul 15 13:06:38.749806 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:38.749753 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" podStartSLOduration=1.749732923 podStartE2EDuration="1.749732923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:38.747844215 +0000 UTC m=+2557.960590726" watchObservedRunningTime="2026-07-15 13:06:38.749732923 +0000 UTC m=+2557.962479433" Jul 15 13:06:48.327068 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:48.327022 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:48.327664 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:48.327108 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:48.339638 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:48.339608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:06:48.789125 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:06:48.789093 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:07:19.224276 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:19.224238 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg"] Jul 15 13:07:34.245418 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.245350 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" podUID="aba9f722-a00a-4668-aa7a-609eddd46bb4" containerName="main" containerID="cri-o://8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4" gracePeriod=45 Jul 15 13:07:34.500721 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.500644 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:07:34.628510 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628471 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9f722-a00a-4668-aa7a-609eddd46bb4-tls-certs\") pod \"aba9f722-a00a-4668-aa7a-609eddd46bb4\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " Jul 15 13:07:34.628729 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628522 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c254\" (UniqueName: \"kubernetes.io/projected/aba9f722-a00a-4668-aa7a-609eddd46bb4-kube-api-access-4c254\") pod \"aba9f722-a00a-4668-aa7a-609eddd46bb4\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " Jul 15 13:07:34.628729 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628551 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-tmp-dir\") pod \"aba9f722-a00a-4668-aa7a-609eddd46bb4\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " Jul 15 13:07:34.628729 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628618 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-model-cache\") pod \"aba9f722-a00a-4668-aa7a-609eddd46bb4\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " Jul 15 13:07:34.628729 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628665 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-dshm\") pod \"aba9f722-a00a-4668-aa7a-609eddd46bb4\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " Jul 15 13:07:34.628950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628743 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-home\") pod \"aba9f722-a00a-4668-aa7a-609eddd46bb4\" (UID: \"aba9f722-a00a-4668-aa7a-609eddd46bb4\") " Jul 15 13:07:34.628950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628876 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aba9f722-a00a-4668-aa7a-609eddd46bb4" (UID: "aba9f722-a00a-4668-aa7a-609eddd46bb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:07:34.628950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.628897 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "aba9f722-a00a-4668-aa7a-609eddd46bb4" (UID: "aba9f722-a00a-4668-aa7a-609eddd46bb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:07:34.629130 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.629090 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-home" (OuterVolumeSpecName: "home") pod "aba9f722-a00a-4668-aa7a-609eddd46bb4" (UID: "aba9f722-a00a-4668-aa7a-609eddd46bb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:07:34.629188 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.629168 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-tmp-dir\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:07:34.629238 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.629192 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-model-cache\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:07:34.630906 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.630886 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba9f722-a00a-4668-aa7a-609eddd46bb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aba9f722-a00a-4668-aa7a-609eddd46bb4" (UID: "aba9f722-a00a-4668-aa7a-609eddd46bb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:07:34.630984 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.630924 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-dshm" (OuterVolumeSpecName: "dshm") pod "aba9f722-a00a-4668-aa7a-609eddd46bb4" (UID: "aba9f722-a00a-4668-aa7a-609eddd46bb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:07:34.630984 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.630969 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba9f722-a00a-4668-aa7a-609eddd46bb4-kube-api-access-4c254" (OuterVolumeSpecName: "kube-api-access-4c254") pod "aba9f722-a00a-4668-aa7a-609eddd46bb4" (UID: "aba9f722-a00a-4668-aa7a-609eddd46bb4"). InnerVolumeSpecName "kube-api-access-4c254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:07:34.730590 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.730522 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9f722-a00a-4668-aa7a-609eddd46bb4-tls-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:07:34.730590 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.730554 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c254\" (UniqueName: \"kubernetes.io/projected/aba9f722-a00a-4668-aa7a-609eddd46bb4-kube-api-access-4c254\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:07:34.730590 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.730599 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-dshm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:07:34.730838 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:34.730612 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9f722-a00a-4668-aa7a-609eddd46bb4-home\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:07:35.010733 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.010692 2582 generic.go:356] "Generic (PLEG): container finished" podID="aba9f722-a00a-4668-aa7a-609eddd46bb4" containerID="8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4" exitCode=0 Jul 15 13:07:35.010950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.010739 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" Jul 15 13:07:35.010950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.010783 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" event={"ID":"aba9f722-a00a-4668-aa7a-609eddd46bb4","Type":"ContainerDied","Data":"8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4"} Jul 15 13:07:35.010950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.010831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg" event={"ID":"aba9f722-a00a-4668-aa7a-609eddd46bb4","Type":"ContainerDied","Data":"cb26d990c4660a9d7d537df9cfd56c4ebe09082a4156f874257fb86c2ed896e5"} Jul 15 13:07:35.010950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.010852 2582 scope.go:117] "RemoveContainer" containerID="8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4" Jul 15 13:07:35.023861 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.023836 2582 scope.go:117] "RemoveContainer" containerID="8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4" Jul 15 13:07:35.024185 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:07:35.024165 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4\": container with ID starting with 8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4 not found: ID does not exist" containerID="8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4" Jul 15 13:07:35.024259 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.024194 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4"} err="failed to get container status \"8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4\": rpc error: code = NotFound desc = could not find container \"8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4\": container with ID starting with 8d13bf0fea47550da70c829730f319ecf799c11cda30cc5794ba1fc3b8534ca4 not found: ID does not exist" Jul 15 13:07:35.042300 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.042266 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg"] Jul 15 13:07:35.050122 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.050090 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-958618/sub-test-model-958618-kserve-7c58686885-7llgg"] Jul 15 13:07:35.315436 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:35.315346 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba9f722-a00a-4668-aa7a-609eddd46bb4" path="/var/lib/kubelet/pods/aba9f722-a00a-4668-aa7a-609eddd46bb4/volumes" Jul 15 13:07:36.640293 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.640254 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd"] Jul 15 13:07:36.640706 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.640682 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba9f722-a00a-4668-aa7a-609eddd46bb4" containerName="main" Jul 15 13:07:36.640706 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.640695 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba9f722-a00a-4668-aa7a-609eddd46bb4" containerName="main" Jul 15 13:07:36.640786 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.640782 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba9f722-a00a-4668-aa7a-609eddd46bb4" containerName="main" Jul 15 13:07:36.645353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.645331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.649797 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.649768 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"rate-test-model-ce0265-kserve-self-signed-certs\"" Jul 15 13:07:36.649935 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.649774 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"openshift-service-ca.crt\"" Jul 15 13:07:36.649935 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.649776 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"kube-root-ca.crt\"" Jul 15 13:07:36.649935 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.649819 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ce0265\"/\"default-dockercfg-lbnzm\"" Jul 15 13:07:36.655474 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.655450 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd"] Jul 15 13:07:36.753680 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.753627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-home\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.753879 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.753706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-dshm\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.753879 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.753790 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c9c67a-7a54-428b-903c-93483794346c-tls-certs\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.753879 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.753863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkr6\" (UniqueName: \"kubernetes.io/projected/f9c9c67a-7a54-428b-903c-93483794346c-kube-api-access-qmkr6\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.754006 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.753910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-tmp-dir\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.754006 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.753942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-model-cache\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855076 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c9c67a-7a54-428b-903c-93483794346c-tls-certs\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855315 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkr6\" (UniqueName: \"kubernetes.io/projected/f9c9c67a-7a54-428b-903c-93483794346c-kube-api-access-qmkr6\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855315 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-tmp-dir\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855315 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-model-cache\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855315 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-home\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855315 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855292 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-dshm\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855673 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855646 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-tmp-dir\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855744 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-model-cache\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.855744 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.855704 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-home\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.857542 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.857514 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-dshm\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.857783 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.857763 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c9c67a-7a54-428b-903c-93483794346c-tls-certs\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.864545 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.864514 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkr6\" (UniqueName: \"kubernetes.io/projected/f9c9c67a-7a54-428b-903c-93483794346c-kube-api-access-qmkr6\") pod \"rate-test-model-ce0265-kserve-898497dc-wpgpd\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:36.956374 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:36.956251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:37.098003 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:37.097970 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd"] Jul 15 13:07:37.099866 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:07:37.099838 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9c9c67a_7a54_428b_903c_93483794346c.slice/crio-4f5d3dd95bf51746a008e4e85d4984992a2612527821b3512b1efad17807561f WatchSource:0}: Error finding container 4f5d3dd95bf51746a008e4e85d4984992a2612527821b3512b1efad17807561f: Status 404 returned error can't find the container with id 4f5d3dd95bf51746a008e4e85d4984992a2612527821b3512b1efad17807561f Jul 15 13:07:38.027763 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:38.027715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" event={"ID":"f9c9c67a-7a54-428b-903c-93483794346c","Type":"ContainerStarted","Data":"4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097"} Jul 15 13:07:38.027763 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:38.027772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" event={"ID":"f9c9c67a-7a54-428b-903c-93483794346c","Type":"ContainerStarted","Data":"4f5d3dd95bf51746a008e4e85d4984992a2612527821b3512b1efad17807561f"} Jul 15 13:07:38.057844 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:38.057790 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" podStartSLOduration=2.057772761 podStartE2EDuration="2.057772761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:38.056141288 +0000 UTC m=+2617.268887803" watchObservedRunningTime="2026-07-15 13:07:38.057772761 +0000 UTC m=+2617.270519272" Jul 15 13:07:46.956804 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:46.956755 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:46.957419 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:46.956825 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:46.969356 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:46.969317 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:07:47.086437 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:07:47.086400 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:09:01.448899 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:09:01.448818 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 13:09:01.457747 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:09:01.457718 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6qvz_e06cd6ee-a6f0-4ab1-95e7-f9154551fac2/ovn-acl-logging/0.log" Jul 15 13:09:49.307627 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:09:49.307589 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd"] Jul 15 13:10:04.328678 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.328617 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" podUID="f9c9c67a-7a54-428b-903c-93483794346c" containerName="main" containerID="cri-o://4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097" gracePeriod=45 Jul 15 13:10:04.593926 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.593862 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:10:04.771437 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771397 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-dshm\") pod \"f9c9c67a-7a54-428b-903c-93483794346c\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " Jul 15 13:10:04.771920 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771449 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c9c67a-7a54-428b-903c-93483794346c-tls-certs\") pod \"f9c9c67a-7a54-428b-903c-93483794346c\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " Jul 15 13:10:04.771920 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771530 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-tmp-dir\") pod \"f9c9c67a-7a54-428b-903c-93483794346c\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " Jul 15 13:10:04.771920 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771620 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmkr6\" (UniqueName: \"kubernetes.io/projected/f9c9c67a-7a54-428b-903c-93483794346c-kube-api-access-qmkr6\") pod \"f9c9c67a-7a54-428b-903c-93483794346c\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " Jul 15 13:10:04.771920 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771687 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-model-cache\") pod \"f9c9c67a-7a54-428b-903c-93483794346c\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " Jul 15 13:10:04.771920 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771752 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-home\") pod \"f9c9c67a-7a54-428b-903c-93483794346c\" (UID: \"f9c9c67a-7a54-428b-903c-93483794346c\") " Jul 15 13:10:04.771920 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.771828 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9c9c67a-7a54-428b-903c-93483794346c" (UID: "f9c9c67a-7a54-428b-903c-93483794346c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:04.772296 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.772032 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9c9c67a-7a54-428b-903c-93483794346c" (UID: "f9c9c67a-7a54-428b-903c-93483794346c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:04.772296 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.772051 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-home" (OuterVolumeSpecName: "home") pod "f9c9c67a-7a54-428b-903c-93483794346c" (UID: "f9c9c67a-7a54-428b-903c-93483794346c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:04.772296 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.772223 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-tmp-dir\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:04.772296 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.772247 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-model-cache\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:04.772296 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.772260 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-home\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:04.773841 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.773811 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-dshm" (OuterVolumeSpecName: "dshm") pod "f9c9c67a-7a54-428b-903c-93483794346c" (UID: "f9c9c67a-7a54-428b-903c-93483794346c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:04.773962 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.773847 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c9c67a-7a54-428b-903c-93483794346c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9c9c67a-7a54-428b-903c-93483794346c" (UID: "f9c9c67a-7a54-428b-903c-93483794346c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:10:04.774127 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.774103 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c9c67a-7a54-428b-903c-93483794346c-kube-api-access-qmkr6" (OuterVolumeSpecName: "kube-api-access-qmkr6") pod "f9c9c67a-7a54-428b-903c-93483794346c" (UID: "f9c9c67a-7a54-428b-903c-93483794346c"). InnerVolumeSpecName "kube-api-access-qmkr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:10:04.777790 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.777760 2582 generic.go:356] "Generic (PLEG): container finished" podID="f9c9c67a-7a54-428b-903c-93483794346c" containerID="4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097" exitCode=0 Jul 15 13:10:04.777890 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.777799 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" Jul 15 13:10:04.777890 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.777842 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" event={"ID":"f9c9c67a-7a54-428b-903c-93483794346c","Type":"ContainerDied","Data":"4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097"} Jul 15 13:10:04.777967 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.777890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd" event={"ID":"f9c9c67a-7a54-428b-903c-93483794346c","Type":"ContainerDied","Data":"4f5d3dd95bf51746a008e4e85d4984992a2612527821b3512b1efad17807561f"} Jul 15 13:10:04.777967 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.777911 2582 scope.go:117] "RemoveContainer" containerID="4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097" Jul 15 13:10:04.796488 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.796468 2582 scope.go:117] "RemoveContainer" containerID="4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097" Jul 15 13:10:04.796812 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:10:04.796793 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097\": container with ID starting with 4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097 not found: ID does not exist" containerID="4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097" Jul 15 13:10:04.796873 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.796821 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097"} err="failed to get container status \"4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097\": rpc error: code = NotFound desc = could not find container \"4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097\": container with ID starting with 4b49c2b662485a1823caa9a2775bba0a534f5791cb7c56bb65c2f5ceac925097 not found: ID does not exist" Jul 15 13:10:04.806819 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.806790 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd"] Jul 15 13:10:04.808542 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.808524 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ce0265/rate-test-model-ce0265-kserve-898497dc-wpgpd"] Jul 15 13:10:04.873356 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.873260 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmkr6\" (UniqueName: \"kubernetes.io/projected/f9c9c67a-7a54-428b-903c-93483794346c-kube-api-access-qmkr6\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:04.873356 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.873296 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c9c67a-7a54-428b-903c-93483794346c-dshm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:04.873356 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:04.873312 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c9c67a-7a54-428b-903c-93483794346c-tls-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:05.314204 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:05.314173 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c9c67a-7a54-428b-903c-93483794346c" path="/var/lib/kubelet/pods/f9c9c67a-7a54-428b-903c-93483794346c/volumes" Jul 15 13:10:35.074062 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:35.074028 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn"] Jul 15 13:10:35.074481 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:35.074311 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" containerID="cri-o://c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803" gracePeriod=30 Jul 15 13:10:40.325272 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.325247 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:10:40.410179 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410115 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-token\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410179 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410145 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-socket\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410199 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-credential-socket\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410225 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5w8m\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-kube-api-access-p5w8m\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410248 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-podinfo\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410270 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istiod-ca-cert\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410393 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410293 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-envoy\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410696 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410479 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-data\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410696 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410498 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:40.410696 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410517 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-certs\") pod \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\" (UID: \"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd\") " Jul 15 13:10:40.410696 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410598 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:40.410972 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.410882 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:40.411037 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.411023 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.411096 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.411041 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.411096 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.411059 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.411391 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.411364 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:40.411871 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.411842 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:10:40.412550 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.412518 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:10:40.412672 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.412623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:10:40.413145 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.413129 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:10:40.413724 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.413703 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-kube-api-access-p5w8m" (OuterVolumeSpecName: "kube-api-access-p5w8m") pod "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" (UID: "165bc244-8c8b-4343-89e8-cc0bb4d1e0bd"). InnerVolumeSpecName "kube-api-access-p5w8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:10:40.512271 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.512247 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5w8m\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-kube-api-access-p5w8m\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.512271 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.512269 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.512389 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.512279 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.512389 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.512288 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.512389 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.512296 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.512389 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.512303 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:10:40.960995 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.960961 2582 generic.go:356] "Generic (PLEG): container finished" podID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerID="c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803" exitCode=0 Jul 15 13:10:40.961163 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.961005 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" Jul 15 13:10:40.961163 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.961008 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" event={"ID":"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd","Type":"ContainerDied","Data":"c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803"} Jul 15 13:10:40.961163 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.961123 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn" event={"ID":"165bc244-8c8b-4343-89e8-cc0bb4d1e0bd","Type":"ContainerDied","Data":"e65894f202876c316f6cf3d787f9e08edbeb705184a19e12907f1a854fd33343"} Jul 15 13:10:40.961163 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.961152 2582 scope.go:117] "RemoveContainer" containerID="c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803" Jul 15 13:10:40.970767 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.970748 2582 scope.go:117] "RemoveContainer" containerID="0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444" Jul 15 13:10:40.980596 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.980541 2582 scope.go:117] "RemoveContainer" containerID="c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803" Jul 15 13:10:40.980858 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:10:40.980836 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803\": container with ID starting with c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803 not found: ID does not exist" containerID="c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803" Jul 15 13:10:40.980922 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.980871 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803"} err="failed to get container status \"c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803\": rpc error: code = NotFound desc = could not find container \"c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803\": container with ID starting with c225ef5a5d164d0caaf71a6524466e9401e4a76ef222f278683db1956074c803 not found: ID does not exist" Jul 15 13:10:40.980922 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.980894 2582 scope.go:117] "RemoveContainer" containerID="0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444" Jul 15 13:10:40.981156 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:10:40.981135 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444\": container with ID starting with 0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444 not found: ID does not exist" containerID="0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444" Jul 15 13:10:40.981247 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.981159 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444"} err="failed to get container status \"0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444\": rpc error: code = NotFound desc = could not find container \"0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444\": container with ID starting with 0b7f93250c92537f2aad61aef9c2e2c48516955be113662dd351facc3a1c7444 not found: ID does not exist" Jul 15 13:10:40.985863 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.985836 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn"] Jul 15 13:10:40.989375 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:40.989355 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ce0265-openshift-default-57fbf7874c-kh2dn"] Jul 15 13:10:41.315208 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:10:41.315176 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" path="/var/lib/kubelet/pods/165bc244-8c8b-4343-89e8-cc0bb4d1e0bd/volumes" Jul 15 13:11:01.163258 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:01.163224 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6"] Jul 15 13:11:01.163751 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:01.163532 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" containerID="cri-o://3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116" gracePeriod=30 Jul 15 13:11:06.418067 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.418041 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:11:06.549011 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.548978 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-podinfo\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549223 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549057 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-certs\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549223 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549211 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istiod-ca-cert\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549251 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-token\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549302 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-socket\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549319 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nmhm\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-kube-api-access-2nmhm\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549322 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:06.549353 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549342 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-data\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549656 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549393 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-credential-socket\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549656 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549455 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-envoy\") pod \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\" (UID: \"2d9bf051-e5fc-4da7-9872-d1d2550087bb\") " Jul 15 13:11:06.549656 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549618 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:11:06.549836 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549622 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:06.549897 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549881 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.549950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549900 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.549950 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.549912 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.550312 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.550287 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:06.550483 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.550451 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:06.551499 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.551477 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:06.551613 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.551497 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-kube-api-access-2nmhm" (OuterVolumeSpecName: "kube-api-access-2nmhm") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "kube-api-access-2nmhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:06.551673 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.551626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:11:06.551808 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.551791 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d9bf051-e5fc-4da7-9872-d1d2550087bb" (UID: "2d9bf051-e5fc-4da7-9872-d1d2550087bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:06.651358 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.651324 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.651358 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.651353 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmhm\" (UniqueName: \"kubernetes.io/projected/2d9bf051-e5fc-4da7-9872-d1d2550087bb-kube-api-access-2nmhm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.651358 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.651363 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.651595 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.651371 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.651595 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.651379 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:06.651595 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:06.651387 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9bf051-e5fc-4da7-9872-d1d2550087bb-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Jul 15 13:11:07.085695 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.085661 2582 generic.go:356] "Generic (PLEG): container finished" podID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerID="3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116" exitCode=0 Jul 15 13:11:07.085845 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.085717 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" Jul 15 13:11:07.085845 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.085720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" event={"ID":"2d9bf051-e5fc-4da7-9872-d1d2550087bb","Type":"ContainerDied","Data":"3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116"} Jul 15 13:11:07.085845 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.085774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6" event={"ID":"2d9bf051-e5fc-4da7-9872-d1d2550087bb","Type":"ContainerDied","Data":"aaf63605e5b991ea8b1922f0469ba58bf0d1f4db8c79e2323aa7dfadbdb828b0"} Jul 15 13:11:07.085845 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.085803 2582 scope.go:117] "RemoveContainer" containerID="3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116" Jul 15 13:11:07.094892 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.094876 2582 scope.go:117] "RemoveContainer" containerID="0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc" Jul 15 13:11:07.102508 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.102488 2582 scope.go:117] "RemoveContainer" containerID="3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116" Jul 15 13:11:07.102806 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:11:07.102783 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116\": container with ID starting with 3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116 not found: ID does not exist" containerID="3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116" Jul 15 13:11:07.102879 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.102816 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116"} err="failed to get container status \"3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116\": rpc error: code = NotFound desc = could not find container \"3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116\": container with ID starting with 3f3784413575ebfa5ce8f2dd19d113bd87d15a831e8952048f0dcbf88ff76116 not found: ID does not exist" Jul 15 13:11:07.102879 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.102837 2582 scope.go:117] "RemoveContainer" containerID="0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc" Jul 15 13:11:07.103089 ip-10-0-128-235 kubenswrapper[2582]: E0715 13:11:07.103070 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc\": container with ID starting with 0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc not found: ID does not exist" containerID="0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc" Jul 15 13:11:07.103138 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.103096 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc"} err="failed to get container status \"0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc\": rpc error: code = NotFound desc = could not find container \"0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc\": container with ID starting with 0e4f7b4d9eab7a096b3cce207fbae2925a798529836db8bc85b3510a886e13dc not found: ID does not exist" Jul 15 13:11:07.109049 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.109030 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6"] Jul 15 13:11:07.113059 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.113039 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-958618-openshift-default-68d65ccc9f-r7kf6"] Jul 15 13:11:07.315847 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.315818 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" path="/var/lib/kubelet/pods/2d9bf051-e5fc-4da7-9872-d1d2550087bb/volumes" Jul 15 13:11:07.935020 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.934996 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-654f68fd79-tspzl_adb0a621-34a8-4b38-a01d-9571cbffbaf6/copy-manifests/0.log" Jul 15 13:11:07.948908 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:07.948886 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-654f68fd79-tspzl_adb0a621-34a8-4b38-a01d-9571cbffbaf6/manager/0.log" Jul 15 13:11:08.432620 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:08.432599 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fdf86cb69-58vwt_aa9e4c84-7feb-48ed-926b-4ca694fd09b2/maas-api/0.log" Jul 15 13:11:11.767441 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:11.767414 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-g87lb_2586511a-8400-4043-a1f6-9d3a6a41ccee/kuadrant-console-plugin/0.log" Jul 15 13:11:12.233062 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:12.232986 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8gqd6_8c4ceb92-9808-4bd0-b939-09dd7d58f0ba/manager/0.log" Jul 15 13:11:12.918852 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:12.918822 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9797f867b-n5pdc_6a3227da-368e-44f4-a26e-08bc425ed795/kube-auth-proxy/0.log" Jul 15 13:11:13.964208 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:13.964178 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vljgn_129db752-348b-43f2-9bf7-e49cdca15448/main/0.log" Jul 15 13:11:14.199605 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:14.199554 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6cnn4_24f58d3d-01c1-4e83-a3d8-41645e82818e/main/0.log" Jul 15 13:11:22.831403 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:22.831374 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x9hqp_d23692c8-7189-494b-aa94-15feeeda6119/global-pull-secret-syncer/0.log" Jul 15 13:11:22.953400 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:22.953375 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n4qp6_ace04528-3fd0-4707-9668-19ce41a4d1e4/konnectivity-agent/0.log" Jul 15 13:11:22.977315 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:22.977291 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-235.ec2.internal_704cb184d5c204a08f126daf6f37229b/haproxy/0.log" Jul 15 13:11:28.263133 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:28.263100 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-g87lb_2586511a-8400-4043-a1f6-9d3a6a41ccee/kuadrant-console-plugin/0.log" Jul 15 13:11:28.592588 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:28.592485 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8gqd6_8c4ceb92-9808-4bd0-b939-09dd7d58f0ba/manager/0.log" Jul 15 13:11:30.255550 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.255480 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/alertmanager/0.log" Jul 15 13:11:30.283129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.283102 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/config-reloader/0.log" Jul 15 13:11:30.308767 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.308743 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/kube-rbac-proxy-web/0.log" Jul 15 13:11:30.332497 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.332479 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/kube-rbac-proxy/0.log" Jul 15 13:11:30.363521 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.363477 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/kube-rbac-proxy-metric/0.log" Jul 15 13:11:30.404406 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.404382 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/prom-label-proxy/0.log" Jul 15 13:11:30.432336 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.432309 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7873b582-119c-4b6a-b68e-4238029dd533/init-config-reloader/0.log" Jul 15 13:11:30.515122 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.515085 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dc75v_31d43a4a-fded-403e-8ba1-c6c2943e4dee/kube-state-metrics/0.log" Jul 15 13:11:30.538741 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.538707 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dc75v_31d43a4a-fded-403e-8ba1-c6c2943e4dee/kube-rbac-proxy-main/0.log" Jul 15 13:11:30.564889 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.564870 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dc75v_31d43a4a-fded-403e-8ba1-c6c2943e4dee/kube-rbac-proxy-self/0.log" Jul 15 13:11:30.608080 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.608053 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bc55c645c-qt8fh_a6790754-343b-45c7-bbda-f664360f8d96/metrics-server/0.log" Jul 15 13:11:30.637886 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.637861 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-mkhdf_3fa1abfd-d881-4cfe-80ce-b6892e879c84/monitoring-plugin/0.log" Jul 15 13:11:30.669819 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.669791 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d524z_99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1/node-exporter/0.log" Jul 15 13:11:30.693617 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.693595 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d524z_99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1/kube-rbac-proxy/0.log" Jul 15 13:11:30.718291 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.718272 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d524z_99a7d5ee-0d6a-4b8f-92af-762f25ce7bf1/init-textfile/0.log" Jul 15 13:11:30.909613 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.909587 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tkkn7_772fd62e-7c74-46d2-8907-0e822a22b6f1/kube-rbac-proxy-main/0.log" Jul 15 13:11:30.932158 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.932129 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tkkn7_772fd62e-7c74-46d2-8907-0e822a22b6f1/kube-rbac-proxy-self/0.log" Jul 15 13:11:30.956195 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:30.956160 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tkkn7_772fd62e-7c74-46d2-8907-0e822a22b6f1/openshift-state-metrics/0.log" Jul 15 13:11:31.001951 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.001927 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/prometheus/0.log" Jul 15 13:11:31.025328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.025306 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/config-reloader/0.log" Jul 15 13:11:31.050698 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.050660 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/thanos-sidecar/0.log" Jul 15 13:11:31.075500 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.075427 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/kube-rbac-proxy-web/0.log" Jul 15 13:11:31.100706 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.100683 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/kube-rbac-proxy/0.log" Jul 15 13:11:31.136286 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.136258 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/kube-rbac-proxy-thanos/0.log" Jul 15 13:11:31.165005 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.164978 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f31fc10-6a7f-4d8d-bcc8-05a08b776761/init-config-reloader/0.log" Jul 15 13:11:31.210800 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.210772 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-5znvn_cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c/prometheus-operator/0.log" Jul 15 13:11:31.248908 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.248883 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-5znvn_cdb9385e-2f4c-4b07-9e32-d75ee6ceda4c/kube-rbac-proxy/0.log" Jul 15 13:11:31.288278 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.288251 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-htsr7_5fab19c5-968b-4452-af0d-fcb562c57f43/prometheus-operator-admission-webhook/0.log" Jul 15 13:11:31.325784 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.325697 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766c6b5949-xpqw5_3395992c-8a59-4e37-8de7-6d760bf93d7f/telemeter-client/0.log" Jul 15 13:11:31.356298 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.356272 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766c6b5949-xpqw5_3395992c-8a59-4e37-8de7-6d760bf93d7f/reload/0.log" Jul 15 13:11:31.404135 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.404104 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766c6b5949-xpqw5_3395992c-8a59-4e37-8de7-6d760bf93d7f/kube-rbac-proxy/0.log" Jul 15 13:11:31.457932 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.457903 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b6b6f9d9c-59h2l_dc63c38d-1ab0-4386-89a2-11cda137d06c/thanos-query/0.log" Jul 15 13:11:31.508969 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.508936 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b6b6f9d9c-59h2l_dc63c38d-1ab0-4386-89a2-11cda137d06c/kube-rbac-proxy-web/0.log" Jul 15 13:11:31.561862 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.561833 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b6b6f9d9c-59h2l_dc63c38d-1ab0-4386-89a2-11cda137d06c/kube-rbac-proxy/0.log" Jul 15 13:11:31.602134 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.602071 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b6b6f9d9c-59h2l_dc63c38d-1ab0-4386-89a2-11cda137d06c/prom-label-proxy/0.log" Jul 15 13:11:31.630466 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.630443 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b6b6f9d9c-59h2l_dc63c38d-1ab0-4386-89a2-11cda137d06c/kube-rbac-proxy-rules/0.log" Jul 15 13:11:31.655840 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:31.655820 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b6b6f9d9c-59h2l_dc63c38d-1ab0-4386-89a2-11cda137d06c/kube-rbac-proxy-metrics/0.log" Jul 15 13:11:32.915924 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.915888 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb"] Jul 15 13:11:32.916311 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916297 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c9c67a-7a54-428b-903c-93483794346c" containerName="main" Jul 15 13:11:32.916311 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916312 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c9c67a-7a54-428b-903c-93483794346c" containerName="main" Jul 15 13:11:32.916394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916320 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" Jul 15 13:11:32.916394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916326 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" Jul 15 13:11:32.916394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916340 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" Jul 15 13:11:32.916394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916345 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" Jul 15 13:11:32.916394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916353 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" Jul 15 13:11:32.916394 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916358 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" Jul 15 13:11:32.916625 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916435 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9bf051-e5fc-4da7-9872-d1d2550087bb" containerName="istio-proxy" Jul 15 13:11:32.916625 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916445 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" Jul 15 13:11:32.916625 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916450 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="165bc244-8c8b-4343-89e8-cc0bb4d1e0bd" containerName="istio-proxy" Jul 15 13:11:32.916625 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.916458 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c9c67a-7a54-428b-903c-93483794346c" containerName="main" Jul 15 13:11:32.919843 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.919819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:32.923059 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.923040 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nr7q\"/\"kube-root-ca.crt\"" Jul 15 13:11:32.924328 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.924302 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nr7q\"/\"openshift-service-ca.crt\"" Jul 15 13:11:32.924428 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.924302 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nr7q\"/\"default-dockercfg-xwh4g\"" Jul 15 13:11:32.941634 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.941610 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb"] Jul 15 13:11:32.984119 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.984090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grsb4\" (UniqueName: \"kubernetes.io/projected/b21e8161-346e-465c-a8f9-edfcb3385bca-kube-api-access-grsb4\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:32.984255 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.984129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-lib-modules\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:32.984255 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.984147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-podres\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:32.984255 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.984215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-sys\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:32.984255 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:32.984252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-proc\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085129 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-proc\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grsb4\" (UniqueName: \"kubernetes.io/projected/b21e8161-346e-465c-a8f9-edfcb3385bca-kube-api-access-grsb4\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085187 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-lib-modules\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-podres\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085297 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-proc\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085506 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-sys\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085506 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085344 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-podres\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085506 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-lib-modules\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.085506 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.085478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b21e8161-346e-465c-a8f9-edfcb3385bca-sys\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.095940 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.095921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grsb4\" (UniqueName: \"kubernetes.io/projected/b21e8161-346e-465c-a8f9-edfcb3385bca-kube-api-access-grsb4\") pod \"perf-node-gather-daemonset-hgsqb\" (UID: \"b21e8161-346e-465c-a8f9-edfcb3385bca\") " pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.230260 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.230196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:33.364673 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:33.364648 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb"] Jul 15 13:11:33.367365 ip-10-0-128-235 kubenswrapper[2582]: W0715 13:11:33.367335 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb21e8161_346e_465c_a8f9_edfcb3385bca.slice/crio-cbe8586c1be63f20a4c11e08adee618b87699bd16116ff08c46b70333f48cdf0 WatchSource:0}: Error finding container cbe8586c1be63f20a4c11e08adee618b87699bd16116ff08c46b70333f48cdf0: Status 404 returned error can't find the container with id cbe8586c1be63f20a4c11e08adee618b87699bd16116ff08c46b70333f48cdf0 Jul 15 13:11:34.215211 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:34.215176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" event={"ID":"b21e8161-346e-465c-a8f9-edfcb3385bca","Type":"ContainerStarted","Data":"1c5b24ac337f62c4ab138e1d64eb721cbc2ade552fee55a88168c47d474f46a7"} Jul 15 13:11:34.215675 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:34.215238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" event={"ID":"b21e8161-346e-465c-a8f9-edfcb3385bca","Type":"ContainerStarted","Data":"cbe8586c1be63f20a4c11e08adee618b87699bd16116ff08c46b70333f48cdf0"} Jul 15 13:11:34.215675 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:34.215287 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:34.243606 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:34.243544 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" podStartSLOduration=2.243529564 podStartE2EDuration="2.243529564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:34.243289739 +0000 UTC m=+2853.456036250" watchObservedRunningTime="2026-07-15 13:11:34.243529564 +0000 UTC m=+2853.456276074" Jul 15 13:11:35.727961 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:35.727928 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6tq6s_55f19759-48e5-44e9-94bd-5bdc03461b51/dns/0.log" Jul 15 13:11:35.752683 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:35.752650 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6tq6s_55f19759-48e5-44e9-94bd-5bdc03461b51/kube-rbac-proxy/0.log" Jul 15 13:11:36.044518 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:36.044492 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2h2z_3df168ff-f781-4feb-a4df-f491a68c3f38/dns-node-resolver/0.log" Jul 15 13:11:36.635817 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:36.635786 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-58gtp_cc03c29a-948a-4119-bb98-f4eb2d8ca13c/node-ca/0.log" Jul 15 13:11:37.815048 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:37.815020 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9797f867b-n5pdc_6a3227da-368e-44f4-a26e-08bc425ed795/kube-auth-proxy/0.log" Jul 15 13:11:38.710943 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:38.710915 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ml5q4_9dcfec8d-a62c-4655-9b8e-2f4da02fb933/serve-healthcheck-canary/0.log" Jul 15 13:11:39.343811 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:39.343779 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9vqh8_8afb2ccd-2367-400d-ac55-e08280a3eaa7/kube-rbac-proxy/0.log" Jul 15 13:11:39.386428 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:39.386403 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9vqh8_8afb2ccd-2367-400d-ac55-e08280a3eaa7/exporter/0.log" Jul 15 13:11:39.421351 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:39.421324 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9vqh8_8afb2ccd-2367-400d-ac55-e08280a3eaa7/extractor/0.log" Jul 15 13:11:40.229809 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:40.229784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nr7q/perf-node-gather-daemonset-hgsqb" Jul 15 13:11:42.340075 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:42.340042 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-654f68fd79-tspzl_adb0a621-34a8-4b38-a01d-9571cbffbaf6/manager/0.log" Jul 15 13:11:42.360364 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:42.360338 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-654f68fd79-tspzl_adb0a621-34a8-4b38-a01d-9571cbffbaf6/copy-manifests/0.log" Jul 15 13:11:43.592164 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:43.592140 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9797f867b-n5pdc_6a3227da-368e-44f4-a26e-08bc425ed795/kube-auth-proxy/0.log" Jul 15 13:11:43.795509 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:43.795475 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-849c5974c6-xbvnz_929be5b8-630d-4a30-a065-3e89fee67d3a/manager/0.log" Jul 15 13:11:45.082291 ip-10-0-128-235 kubenswrapper[2582]: I0715 13:11:45.082260 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9797f867b-n5pdc_6a3227da-368e-44f4-a26e-08bc425ed795/kube-auth-proxy/0.log"