Sep 08 18:34:00.207019 ip-10-0-134-251 systemd[1]: Starting Kubernetes Kubelet... Sep 08 18:34:00.621228 ip-10-0-134-251 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 18:34:00.621228 ip-10-0-134-251 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 18:34:00.621228 ip-10-0-134-251 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 18:34:00.621228 ip-10-0-134-251 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 18:34:00.621228 ip-10-0-134-251 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 18:34:00.623619 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.623543 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 18:34:00.629597 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629573 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:00.629597 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629595 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:00.629703 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629614 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:00.629735 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629720 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:00.629735 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629725 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:00.629735 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629728 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:00.629735 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629731 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:00.629735 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629735 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:00.629735 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629752 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629756 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629759 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629762 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629765 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629768 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629771 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629776 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629780 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629783 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629786 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629789 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629792 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629794 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629797 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629800 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629802 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629805 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629807 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629810 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:00.629907 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629812 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629815 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629817 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629820 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629822 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629825 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629828 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629832 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629835 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629837 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629840 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629842 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629845 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629847 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629854 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629858 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629861 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629864 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629867 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:00.630391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629869 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629872 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629874 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629877 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629880 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629883 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629885 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629888 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629891 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629893 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629896 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629898 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629901 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629904 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629906 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629909 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629911 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629915 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629919 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629922 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:00.630910 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629925 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629927 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629930 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629932 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629935 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629937 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629940 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629947 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629950 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629953 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629956 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629958 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629961 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629964 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629966 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629969 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629972 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629974 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629977 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629980 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:00.631391 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629982 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629985 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629988 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629990 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629993 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629995 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.629998 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630000 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630003 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630005 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630008 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630011 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630711 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630718 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630721 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630724 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630727 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630730 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630733 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630736 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:00.631882 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630758 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630761 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630765 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630769 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630772 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630775 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630778 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630781 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630784 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630788 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630791 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630793 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630796 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630798 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630801 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630804 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630806 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630809 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630811 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630814 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:00.632369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630816 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630819 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630821 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630825 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630828 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630830 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630833 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630835 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630838 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630840 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630843 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630846 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630849 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630851 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630854 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630856 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630860 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630862 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630865 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630868 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:00.632872 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630871 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630873 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630878 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630881 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630884 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630887 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630890 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630893 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630895 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630899 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630902 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630905 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630909 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630913 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630916 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630918 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630921 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630924 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630926 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:00.633369 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630929 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630931 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630934 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630936 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630939 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630942 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630944 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630947 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630950 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630952 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630955 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630957 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630960 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630963 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630966 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630968 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630971 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630974 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630976 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630979 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:00.633840 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630981 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630983 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630986 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630989 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630991 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630993 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630996 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.630998 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631001 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631003 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631006 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631008 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631350 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631355 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631357 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631360 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631364 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631367 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631369 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:00.634327 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631372 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631375 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631377 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631380 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631382 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631385 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631388 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631390 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631394 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631397 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631400 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631403 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631406 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631408 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631411 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631413 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631416 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631418 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631421 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631423 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631426 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:00.634816 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631428 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631431 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631433 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631436 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631438 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631440 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631445 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631449 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631453 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631456 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631459 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631462 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631464 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631467 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631470 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631473 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631476 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631479 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631482 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:00.635331 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631484 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631487 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631490 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631493 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631496 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631498 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631500 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631503 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631505 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631508 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631510 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631513 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631516 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631519 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631522 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631525 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631527 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631530 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631532 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631535 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:00.635827 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631537 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631541 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631544 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631546 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631549 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631551 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631554 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631556 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631559 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631561 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631564 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631568 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631570 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631573 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631575 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631579 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631582 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631584 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631587 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:00.636342 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631589 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631592 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631594 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631597 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631600 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631603 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631606 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631609 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631612 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631614 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631617 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631620 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.631622 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631689 2557 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631725 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631731 2557 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631735 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631754 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631758 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631762 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631766 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 18:34:00.636815 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631769 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631773 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631777 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631780 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631783 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631786 2557 flags.go:64] FLAG: --cgroup-root="" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631789 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631791 2557 flags.go:64] FLAG: --client-ca-file="" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631795 2557 flags.go:64] FLAG: --cloud-provider="external" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631798 2557 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631801 2557 flags.go:64] FLAG: --cluster-domain="" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631804 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631807 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631810 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631813 2557 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631817 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631820 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631822 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631826 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631829 2557 flags.go:64] FLAG: --contention-profiling="false" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631832 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631834 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631837 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631840 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 18:34:00.637319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631844 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631848 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631851 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631853 2557 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631856 2557 flags.go:64] FLAG: --enable-server="true" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631859 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631863 2557 flags.go:64] FLAG: --event-burst="100" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631866 2557 flags.go:64] FLAG: --event-qps="50" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631869 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631873 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631876 2557 flags.go:64] FLAG: --eviction-hard="" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631879 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631882 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631885 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631888 2557 flags.go:64] FLAG: --eviction-soft="" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631891 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631894 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631901 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631904 2557 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631907 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631909 2557 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631912 2557 flags.go:64] FLAG: --feature-gates="" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631915 2557 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631918 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631921 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 18:34:00.637954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631924 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631927 2557 flags.go:64] FLAG: --healthz-port="10248" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631930 2557 flags.go:64] FLAG: --help="false" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631933 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631936 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631939 2557 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631941 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631945 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631949 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631952 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631955 2557 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631957 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631960 2557 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631963 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631966 2557 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631969 2557 flags.go:64] FLAG: --kube-reserved="" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631973 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631975 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631979 2557 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631982 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631984 2557 flags.go:64] FLAG: --lock-file="" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631987 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631990 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631993 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 18:34:00.638574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.631998 2557 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632001 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632004 2557 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632006 2557 flags.go:64] FLAG: --logging-format="text" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632009 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632012 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632015 2557 flags.go:64] FLAG: --manifest-url="" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632018 2557 flags.go:64] FLAG: --manifest-url-header="" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632022 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632032 2557 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632036 2557 flags.go:64] FLAG: --max-pods="110" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632039 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632042 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632045 2557 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632048 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632050 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632056 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632059 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632065 2557 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632068 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632071 2557 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632074 2557 flags.go:64] FLAG: --pod-cidr="" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632077 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 18:34:00.639176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632082 2557 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632086 2557 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632089 2557 flags.go:64] FLAG: --pods-per-core="0" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632092 2557 flags.go:64] FLAG: --port="10250" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632094 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632097 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f87bc67b0ff5494f" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632100 2557 flags.go:64] FLAG: --qos-reserved="" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632103 2557 flags.go:64] FLAG: --read-only-port="10255" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632106 2557 flags.go:64] FLAG: --register-node="true" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632109 2557 flags.go:64] FLAG: --register-with-taints="" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632112 2557 flags.go:64] FLAG: --registry-burst="10" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632116 2557 flags.go:64] FLAG: --registry-qps="5" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632118 2557 flags.go:64] FLAG: --reserved-cpus="" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632121 2557 flags.go:64] FLAG: --reserved-memory="" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632125 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632127 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632130 2557 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632133 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632136 2557 flags.go:64] FLAG: --runonce="false" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632139 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632142 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632144 2557 flags.go:64] FLAG: --seccomp-default="false" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632147 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632150 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632153 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632156 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 18:34:00.639986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632160 2557 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632163 2557 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632166 2557 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632169 2557 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632172 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632175 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632177 2557 flags.go:64] FLAG: --system-cgroups="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632181 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632186 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632189 2557 flags.go:64] FLAG: --tls-cert-file="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632192 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632195 2557 flags.go:64] FLAG: --tls-min-version="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632198 2557 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632201 2557 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632204 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632206 2557 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632209 2557 flags.go:64] FLAG: --v="2" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632215 2557 flags.go:64] FLAG: --version="false" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632219 2557 flags.go:64] FLAG: --vmodule="" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632223 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.632226 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632340 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632345 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632348 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:00.641113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632351 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632354 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632356 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632359 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632362 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632364 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632367 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632370 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632374 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632377 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632379 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632382 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632385 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632387 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632390 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632394 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632397 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632400 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632402 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:00.642173 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632405 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632408 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632410 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632413 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632415 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632418 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632420 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632425 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632429 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632432 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632435 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632437 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632440 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632442 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632445 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632447 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632450 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632452 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632455 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:00.643188 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632458 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632462 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632466 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632469 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632472 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632475 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632477 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632480 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632482 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632487 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632489 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632492 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632494 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632497 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632500 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632502 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632504 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632507 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632509 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632512 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:00.643877 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632514 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632517 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632520 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632522 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632525 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632527 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632530 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632532 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632535 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632537 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632540 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632542 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632545 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632547 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632551 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632554 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632556 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632559 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632561 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632564 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:00.644455 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632566 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632570 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632573 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632576 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632578 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632581 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632583 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632586 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632588 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632591 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632594 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632597 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632599 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632602 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632605 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632609 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632612 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:00.645041 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.632615 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.633683 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.642207 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.642226 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.642251 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.642265 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642347 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642356 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642361 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642365 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642370 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642375 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642379 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642383 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642388 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:00.645472 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642392 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642396 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642400 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642404 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642408 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642413 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642418 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642422 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642426 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642430 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642437 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642443 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642448 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642452 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642457 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642461 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642465 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642469 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642476 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:00.646113 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642481 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642488 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642493 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642497 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642502 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642507 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642511 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642515 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642520 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642524 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642528 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642532 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642536 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642541 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642545 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642549 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642553 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642557 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642561 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:00.646785 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642565 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642569 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642573 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642577 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642581 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642585 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642589 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642594 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642598 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642602 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642606 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642610 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642614 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642618 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642624 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642628 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642633 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642637 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642641 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:00.647392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642645 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642649 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642653 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642657 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642661 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642666 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642670 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642674 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642678 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642682 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642686 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642690 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642694 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642698 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642702 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642706 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642710 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642715 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642719 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642723 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:00.647953 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642727 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642731 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642736 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642756 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642760 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642764 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642778 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642782 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642789 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642795 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642799 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642803 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642807 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.642814 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642976 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:00.648496 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642983 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642988 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642993 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.642998 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643002 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643006 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643013 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643019 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643023 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643028 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643033 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643038 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643042 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643047 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643051 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643055 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643059 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643063 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643067 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:00.648904 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643072 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643078 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643084 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643089 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643094 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643098 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643103 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643107 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643112 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643116 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643121 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643125 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643129 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643133 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643137 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643143 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643149 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643153 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643157 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643161 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:00.649409 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643165 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643169 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643174 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643178 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643182 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643185 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643190 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643194 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643197 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643201 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643205 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643209 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643213 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643217 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643221 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643225 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643229 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643234 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643238 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643242 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:00.649970 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643246 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643251 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643255 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643259 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643263 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643267 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643271 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643276 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643280 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643284 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643288 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643292 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643296 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643300 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643305 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643309 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643313 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643317 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643321 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643325 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:00.650454 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643329 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643333 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643337 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643341 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643345 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643350 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643353 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643357 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643361 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643365 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643369 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643374 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643377 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643381 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643386 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643390 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643395 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643399 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:00.651204 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:00.643402 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.643410 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.643550 2557 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.647224 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.648256 2557 server.go:1019] "Starting client certificate rotation" Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.648353 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.649100 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 18:34:00.651968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.651221 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 18:34:00.652690 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.652662 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 18:34:00.664939 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.664922 2557 log.go:25] "Validated CRI v1 runtime API" Sep 08 18:34:00.668245 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.668231 2557 log.go:25] "Validated CRI v1 image API" Sep 08 18:34:00.669143 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.669128 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 18:34:00.673461 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.673443 2557 fs.go:135] Filesystem UUIDs: map[0aba8e3d-84a0-4dba-b3a9-d00fd339847f:/dev/nvme0n1p4 2ffcddb5-5d8b-4076-8a14-bc0e5bc8b5dd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 08 18:34:00.673542 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.673460 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 18:34:00.675522 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.675501 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 18:34:00.678526 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.678412 2557 manager.go:216] Machine: {Timestamp:2026-09-08 18:34:00.677037899 +0000 UTC m=+0.364094177 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094014 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d6a580f7bffb60151d5148addf57d SystemUUID:ec2d6a58-0f7b-ffb6-0151-d5148addf57d BootID:4732d216-65f1-4db7-bb6b-0af4fd701daa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:a2:c1:4a:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:a2:c1:4a:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:38:a9:a0:e8:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 18:34:00.678526 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.678520 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 18:34:00.678677 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.678606 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 18:34:00.683022 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.683003 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 18:34:00.683144 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.683025 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 18:34:00.683911 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.683900 2557 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 18:34:00.683949 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.683913 2557 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 18:34:00.683949 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.683920 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 18:34:00.683949 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.683932 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 18:34:00.685567 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.685557 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 18:34:00.690769 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690758 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 08 18:34:00.690872 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690863 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 18:34:00.690909 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690890 2557 kubelet.go:486] "Attempting to sync node with API server" Sep 08 18:34:00.690909 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690899 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 18:34:00.690970 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690913 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 18:34:00.690970 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690922 2557 kubelet.go:398] "Adding apiserver pod source" Sep 08 18:34:00.690970 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.690933 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 18:34:00.691222 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.691211 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 18:34:00.691261 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.691252 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 18:34:00.692617 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.692598 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bf5hh" Sep 08 18:34:00.693519 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.693503 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 18:34:00.696298 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696284 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 18:34:00.696357 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696314 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 18:34:00.696471 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696462 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 18:34:00.696505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696477 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 18:34:00.696505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696483 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 18:34:00.696505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696489 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 18:34:00.696505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696497 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 18:34:00.696505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.696503 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 18:34:00.697095 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.697086 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 18:34:00.697095 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.697095 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 18:34:00.697165 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.697101 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 18:34:00.697165 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.697106 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 18:34:00.697165 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.697115 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 18:34:00.697165 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.697123 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 18:34:00.698784 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.698771 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 18:34:00.698784 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.698780 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 18:34:00.699785 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.699763 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bf5hh" Sep 08 18:34:00.701315 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.701295 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 18:34:00.701315 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.701302 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 18:34:00.701422 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.701371 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 18:34:00.702893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.702881 2557 server.go:1268] "Started kubelet" Sep 08 18:34:00.703041 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.702985 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 18:34:00.703132 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.703018 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 18:34:00.703132 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.703080 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 18:34:00.704460 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.704439 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 18:34:00.704936 ip-10-0-134-251 systemd[1]: Started Kubernetes Kubelet. Sep 08 18:34:00.709450 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.709432 2557 server.go:310] "Adding debug handlers to kubelet server" Sep 08 18:34:00.709524 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.709456 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 18:34:00.710829 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.710811 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 18:34:00.710912 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.710850 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 18:34:00.711039 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.711015 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:00.711619 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.711600 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 18:34:00.711619 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.711600 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 18:34:00.711763 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.711630 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 18:34:00.711828 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.711804 2557 reconstruct.go:97] "Volume reconstruction finished" Sep 08 18:34:00.711828 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.711815 2557 reconciler.go:29] "Reconciler: start to sync state" Sep 08 18:34:00.712304 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712286 2557 factory.go:55] Registering systemd factory Sep 08 18:34:00.712304 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712306 2557 factory.go:223] Registration of the systemd container factory successfully Sep 08 18:34:00.712488 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712477 2557 factory.go:153] Registering CRI-O factory Sep 08 18:34:00.712488 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712489 2557 factory.go:223] Registration of the crio container factory successfully Sep 08 18:34:00.712598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712536 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 18:34:00.712598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712550 2557 factory.go:103] Registering Raw factory Sep 08 18:34:00.712598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712563 2557 manager.go:1196] Started watching for new ooms in manager Sep 08 18:34:00.712974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712957 2557 manager.go:349] Starting recovery of all containers Sep 08 18:34:00.712974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.712973 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:00.715463 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.715440 2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-251.ec2.internal\" not found" node="ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.722905 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.722891 2557 manager.go:354] Recovery completed Sep 08 18:34:00.727008 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.726997 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.728602 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.728585 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.728663 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.728616 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.728663 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.728634 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.729558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.729540 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 18:34:00.729558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.729556 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 18:34:00.729682 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.729555 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.729682 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.729575 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 08 18:34:00.730237 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.730223 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.730284 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.730247 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.730284 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.730260 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.731859 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.731840 2557 policy_none.go:49] "None policy: Start" Sep 08 18:34:00.731859 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.731856 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 18:34:00.731998 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.731871 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 18:34:00.733834 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.733811 2557 policy_none.go:47] "Start" Sep 08 18:34:00.759424 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759408 2557 manager.go:337] "Starting Device Plugin manager" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.759447 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759460 2557 server.go:87] "Starting device plugin registration server" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759655 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759668 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759828 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759906 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.759922 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.760380 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.760432 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.839266 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.840502 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.840555 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.840578 2557 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.840623 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.842637 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.860401 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.861158 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.861181 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.861190 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.861209 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.870284 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.882011 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.870299 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-251.ec2.internal\": node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:00.882632 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.882304 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:00.941577 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.941552 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal"] Sep 08 18:34:00.941628 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.941624 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.942775 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.942762 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.942843 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.942783 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.942843 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.942794 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.945075 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945064 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.945189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945176 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.945248 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945202 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.945700 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945688 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.945771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945699 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.945771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945710 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.945771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945721 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.945771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945723 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.945771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.945734 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.947990 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.947977 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.948039 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.948001 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:00.948598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.948582 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:00.948666 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.948607 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:00.948666 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:00.948618 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:00.967242 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.967225 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-251.ec2.internal\" not found" node="ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.971541 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.971526 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-251.ec2.internal\" not found" node="ip-10-0-134-251.ec2.internal" Sep 08 18:34:00.983195 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:00.983179 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:01.084219 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:01.084199 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:01.113558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.113542 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a3853aaaa6ace0991cc2a038a9653ab-config\") pod \"kube-apiserver-proxy-ip-10-0-134-251.ec2.internal\" (UID: \"5a3853aaaa6ace0991cc2a038a9653ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.113647 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.113570 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14f96a47f3d21417de7f2c2d7e4702ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal\" (UID: \"14f96a47f3d21417de7f2c2d7e4702ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.113647 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.113595 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14f96a47f3d21417de7f2c2d7e4702ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal\" (UID: \"14f96a47f3d21417de7f2c2d7e4702ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.184808 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:01.184765 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:01.214225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.214208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14f96a47f3d21417de7f2c2d7e4702ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal\" (UID: \"14f96a47f3d21417de7f2c2d7e4702ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.214279 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.214229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14f96a47f3d21417de7f2c2d7e4702ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal\" (UID: \"14f96a47f3d21417de7f2c2d7e4702ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.214279 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.214245 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a3853aaaa6ace0991cc2a038a9653ab-config\") pod \"kube-apiserver-proxy-ip-10-0-134-251.ec2.internal\" (UID: \"5a3853aaaa6ace0991cc2a038a9653ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.214350 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.214278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a3853aaaa6ace0991cc2a038a9653ab-config\") pod \"kube-apiserver-proxy-ip-10-0-134-251.ec2.internal\" (UID: \"5a3853aaaa6ace0991cc2a038a9653ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.214350 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.214316 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14f96a47f3d21417de7f2c2d7e4702ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal\" (UID: \"14f96a47f3d21417de7f2c2d7e4702ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.214350 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.214315 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14f96a47f3d21417de7f2c2d7e4702ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal\" (UID: \"14f96a47f3d21417de7f2c2d7e4702ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.269363 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.269338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.273946 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:01.273933 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" Sep 08 18:34:01.285274 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:01.285260 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:01.385816 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:01.385789 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:02.664162 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:02.664141 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:02.763704 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.763685 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:02.764959 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:02.764943 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:02.826988 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.826968 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 18:34:02.827328 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.827096 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 18:34:02.827328 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.827103 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 18:34:02.827328 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.827096 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 18:34:02.865289 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:02.865253 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:02.885557 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.885517 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 18:29:01 +0000 UTC" deadline="2028-02-16 09:48:53.989291763 +0000 UTC" Sep 08 18:34:02.885668 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:02.885559 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12615h14m51.103739568s" Sep 08 18:34:02.966084 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:02.966069 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:03.062487 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:03.062463 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f96a47f3d21417de7f2c2d7e4702ad.slice/crio-95e93c03061eec26dad506704a16c88bd41dc3e86c2118ae497df3c40f095e9c WatchSource:0}: Error finding container 95e93c03061eec26dad506704a16c88bd41dc3e86c2118ae497df3c40f095e9c: Status 404 returned error can't find the container with id 95e93c03061eec26dad506704a16c88bd41dc3e86c2118ae497df3c40f095e9c Sep 08 18:34:03.066264 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:03.066248 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:03.066351 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.066334 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:34:03.142142 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:03.142121 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3853aaaa6ace0991cc2a038a9653ab.slice/crio-1bb9ac4c563360f11ab3b2cdfc3c3b7000d0c0d7896d8dd8861ab0ae89b5321e WatchSource:0}: Error finding container 1bb9ac4c563360f11ab3b2cdfc3c3b7000d0c0d7896d8dd8861ab0ae89b5321e: Status 404 returned error can't find the container with id 1bb9ac4c563360f11ab3b2cdfc3c3b7000d0c0d7896d8dd8861ab0ae89b5321e Sep 08 18:34:03.166464 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:03.166440 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-251.ec2.internal\" not found" Sep 08 18:34:03.228145 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.228100 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:03.289593 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.289573 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" Sep 08 18:34:03.299010 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.298994 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 18:34:03.300120 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.300108 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" Sep 08 18:34:03.305097 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.305085 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 18:34:03.864158 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.864126 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:03.870067 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.870047 2557 apiserver.go:52] "Watching apiserver" Sep 08 18:34:03.876453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.876429 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 18:34:03.876824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.876795 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-75t6m","openshift-network-operator/iptables-alerter-pvmjr","kube-system/konnectivity-agent-mqll6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5","openshift-dns/node-resolver-ntg6k","openshift-image-registry/node-ca-5m7lw","openshift-multus/multus-dkdtb","openshift-ovn-kubernetes/ovnkube-node-ttvkc","kube-system/global-pull-secret-syncer-4c29z","kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal","openshift-cluster-node-tuning-operator/tuned-j9n4r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal","openshift-multus/multus-additional-cni-plugins-z7nvh","openshift-multus/network-metrics-daemon-7rdgz"] Sep 08 18:34:03.879185 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.879161 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.881573 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.881521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.881775 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.881755 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4wgz8\"" Sep 08 18:34:03.881866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.881847 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 18:34:03.881924 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.881854 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.881924 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.881882 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.882028 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.881984 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 18:34:03.884684 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.884295 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 18:34:03.884684 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.884316 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6tr75\"" Sep 08 18:34:03.884684 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.884331 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 18:34:03.884684 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.884348 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:03.886174 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.886155 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.886573 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.886552 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 18:34:03.886677 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.886554 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2c64l\"" Sep 08 18:34:03.886736 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.886558 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 18:34:03.888497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.888478 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 18:34:03.888727 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.888709 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.888818 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.888740 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snghn\"" Sep 08 18:34:03.889132 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.888938 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.889132 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.889075 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 18:34:03.890067 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.889715 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:03.891700 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.891682 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:03.891846 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.891825 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.892171 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.892151 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.892287 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.892273 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x59sx\"" Sep 08 18:34:03.893859 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.893841 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.893944 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.893861 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5tdkq\"" Sep 08 18:34:03.894018 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.894002 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 18:34:03.894070 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.894058 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.894133 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.894117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:03.894213 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:03.894193 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:03.896452 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.896384 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:03.896539 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:03.896449 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.899921 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-daemon-config\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.899953 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-multus-certs\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.899978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.900003 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-socket-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.900030 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.900073 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.900297 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4766c049-9d22-4ff0-865c-d7935eddcda9-agent-certs\") pod \"konnectivity-agent-mqll6\" (UID: \"4766c049-9d22-4ff0-865c-d7935eddcda9\") " pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.900936 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.900967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq24m\" (UniqueName: \"kubernetes.io/projected/2e030b87-5747-4a52-ae8a-1f33fe710f9b-kube-api-access-hq24m\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901019 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-cni-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901047 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-system-cni-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-socket-dir-parent\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901255 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-k8s-cni-cncf-io\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901291 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-conf-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901356 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cnibin\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4766c049-9d22-4ff0-865c-d7935eddcda9-konnectivity-ca\") pod \"konnectivity-agent-mqll6\" (UID: \"4766c049-9d22-4ff0-865c-d7935eddcda9\") " pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:03.901673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-sys-fs\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901538 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wsp\" (UniqueName: \"kubernetes.io/projected/728eba11-c5b4-4f60-afed-b980d81a15e6-kube-api-access-96wsp\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-netns\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w29q5\" (UniqueName: \"kubernetes.io/projected/c65b4c43-1caf-4823-adc2-653bb8ff7a08-kube-api-access-w29q5\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901813 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtz9\" (UniqueName: \"kubernetes.io/projected/340d0037-0c5e-4280-b788-68537331aa79-kube-api-access-gmtz9\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e030b87-5747-4a52-ae8a-1f33fe710f9b-hosts-file\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901912 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/728eba11-c5b4-4f60-afed-b980d81a15e6-host\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c65b4c43-1caf-4823-adc2-653bb8ff7a08-cni-binary-copy\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.901978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-kubelet\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902313 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-hostroot\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902348 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902381 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-registration-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902412 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/728eba11-c5b4-4f60-afed-b980d81a15e6-serviceca\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902456 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-cni-bin\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.902497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902497 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902562 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-etc-kubernetes\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.902929 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mf7c\" (UniqueName: \"kubernetes.io/projected/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-kube-api-access-5mf7c\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903054 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-device-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903090 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e030b87-5747-4a52-ae8a-1f33fe710f9b-tmp-dir\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903119 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-cnibin\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.903170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903145 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-cni-multus\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:03.903419 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903196 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.903419 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903232 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8vp94\"" Sep 08 18:34:03.903995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.903965 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 18:34:03.904192 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.904175 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:03.904279 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:03.904253 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:03.904728 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.904709 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.905467 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.905448 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 18:34:03.906736 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.906295 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:03.908595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.908521 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.908595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.908560 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.908838 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.908820 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:03.908882 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.908870 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m7wkk\"" Sep 08 18:34:03.910785 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.910764 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 18:34:03.910950 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.910934 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 18:34:03.911029 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.910984 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 18:34:03.911393 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.911313 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 18:34:03.911455 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.911427 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-44njf\"" Sep 08 18:34:03.911563 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.911533 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 18:34:03.911720 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.911700 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 18:34:03.918985 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.918965 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-89lx5" Sep 08 18:34:03.926045 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.926026 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-89lx5" Sep 08 18:34:03.990896 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:03.990877 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 18:34:04.003923 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.003905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.003923 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.003930 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.004085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.003948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-kubelet-config\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.004085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.003967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-sys\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.004085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004045 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.004085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004078 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-log-socket\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004109 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-system-cni-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004133 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-socket-dir-parent\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004148 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-k8s-cni-cncf-io\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-conf-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004181 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004198 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5br64\" (UniqueName: \"kubernetes.io/projected/04151103-1947-499e-a096-b70859a454ff-kube-api-access-5br64\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004219 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmc8w\" (UniqueName: \"kubernetes.io/projected/34678d45-07b6-4bed-88dd-a083983489f3-kube-api-access-bmc8w\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004230 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-system-cni-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-k8s-cni-cncf-io\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004233 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovnkube-config\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-conf-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cnibin\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4766c049-9d22-4ff0-865c-d7935eddcda9-konnectivity-ca\") pod \"konnectivity-agent-mqll6\" (UID: \"4766c049-9d22-4ff0-865c-d7935eddcda9\") " pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:04.004355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004357 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cnibin\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004373 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-socket-dir-parent\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wsp\" (UniqueName: \"kubernetes.io/projected/728eba11-c5b4-4f60-afed-b980d81a15e6-kube-api-access-96wsp\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004416 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w29q5\" (UniqueName: \"kubernetes.io/projected/c65b4c43-1caf-4823-adc2-653bb8ff7a08-kube-api-access-w29q5\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004467 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-env-overrides\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovn-node-metrics-cert\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004512 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004528 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-node-log\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004554 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/728eba11-c5b4-4f60-afed-b980d81a15e6-host\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004560 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-kubelet\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-kubelet\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-kubelet\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004630 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004669 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/728eba11-c5b4-4f60-afed-b980d81a15e6-host\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004690 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8b0b394-f387-4007-8ef3-46024a45b449-host-slash\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.005225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004794 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysctl-conf\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004827 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-run\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004891 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-lib-modules\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004937 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-var-lib-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.004979 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005015 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-kubernetes\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005059 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-var-lib-kubelet\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005086 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mf7c\" (UniqueName: \"kubernetes.io/projected/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-kube-api-access-5mf7c\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005111 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-device-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005134 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-cni-multus\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005135 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005155 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-daemon-config\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005197 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4766c049-9d22-4ff0-865c-d7935eddcda9-konnectivity-ca\") pod \"konnectivity-agent-mqll6\" (UID: \"4766c049-9d22-4ff0-865c-d7935eddcda9\") " pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005201 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-systemd-units\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005252 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovnkube-script-lib\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005275 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4766c049-9d22-4ff0-865c-d7935eddcda9-agent-certs\") pod \"konnectivity-agent-mqll6\" (UID: \"4766c049-9d22-4ff0-865c-d7935eddcda9\") " pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005302 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-device-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.006049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq24m\" (UniqueName: \"kubernetes.io/projected/2e030b87-5747-4a52-ae8a-1f33fe710f9b-kube-api-access-hq24m\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005349 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-cni-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005376 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-cni-multus\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-etc-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-ovn\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-systemd\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005535 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-sys-fs\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-netns\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-cni-dir\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005581 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-dbus\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005634 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-netns\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005779 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-systemd\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005778 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c65b4c43-1caf-4823-adc2-653bb8ff7a08-multus-daemon-config\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.006854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005894 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-sys-fs\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtz9\" (UniqueName: \"kubernetes.io/projected/340d0037-0c5e-4280-b788-68537331aa79-kube-api-access-gmtz9\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005949 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnhs\" (UniqueName: \"kubernetes.io/projected/e8b0b394-f387-4007-8ef3-46024a45b449-kube-api-access-lmnhs\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.005972 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-host\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34678d45-07b6-4bed-88dd-a083983489f3-tmp\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006059 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-run-netns\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006116 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-cni-bin\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006143 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e030b87-5747-4a52-ae8a-1f33fe710f9b-hosts-file\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c65b4c43-1caf-4823-adc2-653bb8ff7a08-cni-binary-copy\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-hostroot\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006231 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-modprobe-d\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006262 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysconfig\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006285 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-registration-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006293 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e030b87-5747-4a52-ae8a-1f33fe710f9b-hosts-file\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/728eba11-c5b4-4f60-afed-b980d81a15e6-serviceca\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-cni-bin\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006370 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8b0b394-f387-4007-8ef3-46024a45b449-iptables-alerter-script\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.007622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-cni-netd\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006428 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-etc-kubernetes\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006498 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-slash\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006528 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysctl-d\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e030b87-5747-4a52-ae8a-1f33fe710f9b-tmp-dir\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-cnibin\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006614 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c65b4c43-1caf-4823-adc2-653bb8ff7a08-cni-binary-copy\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006641 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-multus-certs\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006684 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34678d45-07b6-4bed-88dd-a083983489f3-etc-tuned\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk57n\" (UniqueName: \"kubernetes.io/projected/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-kube-api-access-qk57n\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006728 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006769 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-hostroot\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006792 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-socket-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006793 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/728eba11-c5b4-4f60-afed-b980d81a15e6-serviceca\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.008428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006846 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-etc-kubernetes\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006868 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-var-lib-cni-bin\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-cnibin\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006795 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.006969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c65b4c43-1caf-4823-adc2-653bb8ff7a08-host-run-multus-certs\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.007195 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-socket-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.007351 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e030b87-5747-4a52-ae8a-1f33fe710f9b-tmp-dir\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.007379 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/340d0037-0c5e-4280-b788-68537331aa79-registration-dir\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.009232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.008208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.010132 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.010110 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4766c049-9d22-4ff0-865c-d7935eddcda9-agent-certs\") pod \"konnectivity-agent-mqll6\" (UID: \"4766c049-9d22-4ff0-865c-d7935eddcda9\") " pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:04.013122 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.013094 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w29q5\" (UniqueName: \"kubernetes.io/projected/c65b4c43-1caf-4823-adc2-653bb8ff7a08-kube-api-access-w29q5\") pod \"multus-dkdtb\" (UID: \"c65b4c43-1caf-4823-adc2-653bb8ff7a08\") " pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.014249 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.014227 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq24m\" (UniqueName: \"kubernetes.io/projected/2e030b87-5747-4a52-ae8a-1f33fe710f9b-kube-api-access-hq24m\") pod \"node-resolver-ntg6k\" (UID: \"2e030b87-5747-4a52-ae8a-1f33fe710f9b\") " pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.014595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.014573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mf7c\" (UniqueName: \"kubernetes.io/projected/2547b1f7-3386-4ba6-ba73-e0efcdfbb57b-kube-api-access-5mf7c\") pod \"multus-additional-cni-plugins-z7nvh\" (UID: \"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b\") " pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.015125 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.015105 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wsp\" (UniqueName: \"kubernetes.io/projected/728eba11-c5b4-4f60-afed-b980d81a15e6-kube-api-access-96wsp\") pod \"node-ca-5m7lw\" (UID: \"728eba11-c5b4-4f60-afed-b980d81a15e6\") " pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.015715 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.015621 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtz9\" (UniqueName: \"kubernetes.io/projected/340d0037-0c5e-4280-b788-68537331aa79-kube-api-access-gmtz9\") pod \"aws-ebs-csi-driver-node-bvws5\" (UID: \"340d0037-0c5e-4280-b788-68537331aa79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.027032 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.026985 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" event={"ID":"5a3853aaaa6ace0991cc2a038a9653ab","Type":"ContainerStarted","Data":"1bb9ac4c563360f11ab3b2cdfc3c3b7000d0c0d7896d8dd8861ab0ae89b5321e"} Sep 08 18:34:04.028229 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.028207 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" event={"ID":"14f96a47f3d21417de7f2c2d7e4702ad","Type":"ContainerStarted","Data":"95e93c03061eec26dad506704a16c88bd41dc3e86c2118ae497df3c40f095e9c"} Sep 08 18:34:04.107923 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.107884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-systemd-units\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.107923 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.107921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovnkube-script-lib\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.107948 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-etc-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.107971 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.107980 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-systemd-units\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.107995 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-ovn\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-systemd\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108026 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-etc-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-dbus\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108054 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-ovn\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108066 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-systemd\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108085 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108091 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnhs\" (UniqueName: \"kubernetes.io/projected/e8b0b394-f387-4007-8ef3-46024a45b449-kube-api-access-lmnhs\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-host\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108135 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34678d45-07b6-4bed-88dd-a083983489f3-tmp\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108144 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-dbus\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108157 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-run-netns\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108178 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-systemd\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108179 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-cni-bin\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108206 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-modprobe-d\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108212 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-host\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysconfig\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-systemd\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108252 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8b0b394-f387-4007-8ef3-46024a45b449-iptables-alerter-script\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-cni-netd\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108300 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-slash\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysctl-d\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108349 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34678d45-07b6-4bed-88dd-a083983489f3-etc-tuned\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108373 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk57n\" (UniqueName: \"kubernetes.io/projected/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-kube-api-access-qk57n\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-kubelet-config\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108430 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-sys\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.108777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-log-socket\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108507 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5br64\" (UniqueName: \"kubernetes.io/projected/04151103-1947-499e-a096-b70859a454ff-kube-api-access-5br64\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108556 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmc8w\" (UniqueName: \"kubernetes.io/projected/34678d45-07b6-4bed-88dd-a083983489f3-kube-api-access-bmc8w\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-cni-bin\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovnkube-config\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-env-overrides\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108742 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovn-node-metrics-cert\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-node-log\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108791 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-kubelet\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108817 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8b0b394-f387-4007-8ef3-46024a45b449-host-slash\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108842 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysctl-conf\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108865 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-run\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108893 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-lib-modules\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-var-lib-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.109629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108943 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108961 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-modprobe-d\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-kubernetes\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-var-lib-kubelet\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109025 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-kubernetes\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109069 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-var-lib-kubelet\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109075 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovnkube-config\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109102 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysconfig\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109139 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-cni-netd\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109140 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-sys\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-slash\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109194 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-run-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109240 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-log-socket\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109263 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysctl-d\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.109324 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109328 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-run-netns\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108528 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovnkube-script-lib\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.110541 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.109402 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:34:04.609360971 +0000 UTC m=+3.118513393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109402 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-node-log\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.108866 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8b0b394-f387-4007-8ef3-46024a45b449-iptables-alerter-script\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8b0b394-f387-4007-8ef3-46024a45b449-host-slash\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109597 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-kubelet-config\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-kubelet\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-var-lib-openvswitch\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109672 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-run\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-etc-sysctl-conf\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109716 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109718 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-env-overrides\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.109762 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34678d45-07b6-4bed-88dd-a083983489f3-lib-modules\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.109818 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:04.111402 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.109885 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:04.609867451 +0000 UTC m=+3.119019875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:04.112005 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.111442 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34678d45-07b6-4bed-88dd-a083983489f3-tmp\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.112005 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.111953 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34678d45-07b6-4bed-88dd-a083983489f3-etc-tuned\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.112149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.112132 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-ovn-node-metrics-cert\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.115435 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.115365 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnhs\" (UniqueName: \"kubernetes.io/projected/e8b0b394-f387-4007-8ef3-46024a45b449-kube-api-access-lmnhs\") pod \"iptables-alerter-pvmjr\" (UID: \"e8b0b394-f387-4007-8ef3-46024a45b449\") " pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.115554 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.115489 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:04.115554 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.115506 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:04.115554 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.115519 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:04.115728 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.115630 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:04.61561471 +0000 UTC m=+3.124767146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:04.117356 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.117306 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk57n\" (UniqueName: \"kubernetes.io/projected/61de7352-8afc-4cb4-b04e-9b1556b7bd6a-kube-api-access-qk57n\") pod \"ovnkube-node-ttvkc\" (UID: \"61de7352-8afc-4cb4-b04e-9b1556b7bd6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.117722 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.117702 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmc8w\" (UniqueName: \"kubernetes.io/projected/34678d45-07b6-4bed-88dd-a083983489f3-kube-api-access-bmc8w\") pod \"tuned-j9n4r\" (UID: \"34678d45-07b6-4bed-88dd-a083983489f3\") " pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.117823 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.117807 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5br64\" (UniqueName: \"kubernetes.io/projected/04151103-1947-499e-a096-b70859a454ff-kube-api-access-5br64\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:04.190625 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.190600 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkdtb" Sep 08 18:34:04.199811 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.199788 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" Sep 08 18:34:04.208366 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.208348 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:04.215951 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.215933 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" Sep 08 18:34:04.228546 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.228523 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ntg6k" Sep 08 18:34:04.235126 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.235108 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5m7lw" Sep 08 18:34:04.243631 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.243614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvmjr" Sep 08 18:34:04.250182 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.250164 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" Sep 08 18:34:04.257877 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.257860 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:04.414140 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.414074 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:04.612641 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.612608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:04.612834 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.612680 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:04.612834 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.612769 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:04.612834 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.612778 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:04.612834 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.612833 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:05.612813078 +0000 UTC m=+4.121965510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:04.613046 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.612851 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:34:05.612841998 +0000 UTC m=+4.121994420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:04.712995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.712922 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:04.713120 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.713091 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:04.713120 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.713110 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:04.713214 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.713123 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:04.713214 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:04.713187 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:05.713161943 +0000 UTC m=+4.222314377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:04.929544 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.929498 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:29:03 +0000 UTC" deadline="2028-04-11 22:13:09.715274957 +0000 UTC" Sep 08 18:34:04.929544 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:04.929535 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13947h39m4.785744522s" Sep 08 18:34:05.019216 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.019193 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728eba11_c5b4_4f60_afed_b980d81a15e6.slice/crio-01dfd20a4c1b3ca62ca29496d4a1aa616214e4a159ec5d95de9af140870f1c71 WatchSource:0}: Error finding container 01dfd20a4c1b3ca62ca29496d4a1aa616214e4a159ec5d95de9af140870f1c71: Status 404 returned error can't find the container with id 01dfd20a4c1b3ca62ca29496d4a1aa616214e4a159ec5d95de9af140870f1c71 Sep 08 18:34:05.020845 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.020825 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e030b87_5747_4a52_ae8a_1f33fe710f9b.slice/crio-9f16f01c0500b488755757f6f2fcd1abffaaf868d19baef7f6ac2399f9110831 WatchSource:0}: Error finding container 9f16f01c0500b488755757f6f2fcd1abffaaf868d19baef7f6ac2399f9110831: Status 404 returned error can't find the container with id 9f16f01c0500b488755757f6f2fcd1abffaaf868d19baef7f6ac2399f9110831 Sep 08 18:34:05.022992 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.022808 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b0b394_f387_4007_8ef3_46024a45b449.slice/crio-93ad82adb9582f6a3b2d25f0db54374f7ae4bc4f442bf5ab8968698ed51bb325 WatchSource:0}: Error finding container 93ad82adb9582f6a3b2d25f0db54374f7ae4bc4f442bf5ab8968698ed51bb325: Status 404 returned error can't find the container with id 93ad82adb9582f6a3b2d25f0db54374f7ae4bc4f442bf5ab8968698ed51bb325 Sep 08 18:34:05.024571 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.024523 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61de7352_8afc_4cb4_b04e_9b1556b7bd6a.slice/crio-3cd53139a9eda7fb570a00fed39e6f30064c98c2e662f499f35289f1d63b4845 WatchSource:0}: Error finding container 3cd53139a9eda7fb570a00fed39e6f30064c98c2e662f499f35289f1d63b4845: Status 404 returned error can't find the container with id 3cd53139a9eda7fb570a00fed39e6f30064c98c2e662f499f35289f1d63b4845 Sep 08 18:34:05.027103 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.026790 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340d0037_0c5e_4280_b788_68537331aa79.slice/crio-63e8de84d52a4b161e620fc7f50a29a3428e9ccd1937c19bc3d7178f4ec39a87 WatchSource:0}: Error finding container 63e8de84d52a4b161e620fc7f50a29a3428e9ccd1937c19bc3d7178f4ec39a87: Status 404 returned error can't find the container with id 63e8de84d52a4b161e620fc7f50a29a3428e9ccd1937c19bc3d7178f4ec39a87 Sep 08 18:34:05.027628 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.027600 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2547b1f7_3386_4ba6_ba73_e0efcdfbb57b.slice/crio-8c26c3912c979959d8185f0cd785190b69d269a98bb5f59e22f89fcd4008d3c1 WatchSource:0}: Error finding container 8c26c3912c979959d8185f0cd785190b69d269a98bb5f59e22f89fcd4008d3c1: Status 404 returned error can't find the container with id 8c26c3912c979959d8185f0cd785190b69d269a98bb5f59e22f89fcd4008d3c1 Sep 08 18:34:05.029235 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.029213 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65b4c43_1caf_4823_adc2_653bb8ff7a08.slice/crio-868937b8a2d6a68d7350d69106866bd1b800e745c8e1a932c0dccfcce5839426 WatchSource:0}: Error finding container 868937b8a2d6a68d7350d69106866bd1b800e745c8e1a932c0dccfcce5839426: Status 404 returned error can't find the container with id 868937b8a2d6a68d7350d69106866bd1b800e745c8e1a932c0dccfcce5839426 Sep 08 18:34:05.030554 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:05.030500 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4766c049_9d22_4ff0_865c_d7935eddcda9.slice/crio-0f6f02d2c973d02f75046f927bf384d37051c7ec368abcbac7166ddbc2b5eafc WatchSource:0}: Error finding container 0f6f02d2c973d02f75046f927bf384d37051c7ec368abcbac7166ddbc2b5eafc: Status 404 returned error can't find the container with id 0f6f02d2c973d02f75046f927bf384d37051c7ec368abcbac7166ddbc2b5eafc Sep 08 18:34:05.031098 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.030909 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerStarted","Data":"8c26c3912c979959d8185f0cd785190b69d269a98bb5f59e22f89fcd4008d3c1"} Sep 08 18:34:05.034895 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.034729 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" event={"ID":"340d0037-0c5e-4280-b788-68537331aa79","Type":"ContainerStarted","Data":"63e8de84d52a4b161e620fc7f50a29a3428e9ccd1937c19bc3d7178f4ec39a87"} Sep 08 18:34:05.036032 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.036008 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"3cd53139a9eda7fb570a00fed39e6f30064c98c2e662f499f35289f1d63b4845"} Sep 08 18:34:05.037370 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.037344 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvmjr" event={"ID":"e8b0b394-f387-4007-8ef3-46024a45b449","Type":"ContainerStarted","Data":"93ad82adb9582f6a3b2d25f0db54374f7ae4bc4f442bf5ab8968698ed51bb325"} Sep 08 18:34:05.039091 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.039068 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ntg6k" event={"ID":"2e030b87-5747-4a52-ae8a-1f33fe710f9b","Type":"ContainerStarted","Data":"9f16f01c0500b488755757f6f2fcd1abffaaf868d19baef7f6ac2399f9110831"} Sep 08 18:34:05.040320 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.040302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5m7lw" event={"ID":"728eba11-c5b4-4f60-afed-b980d81a15e6","Type":"ContainerStarted","Data":"01dfd20a4c1b3ca62ca29496d4a1aa616214e4a159ec5d95de9af140870f1c71"} Sep 08 18:34:05.089628 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.089609 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:05.619640 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.619365 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:05.619813 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.619764 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:05.619875 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.619826 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:34:07.619808538 +0000 UTC m=+6.128960971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:05.619875 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.619842 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:05.619985 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.619905 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:07.619885909 +0000 UTC m=+6.129038333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:05.620257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.620158 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:05.721195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.721164 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:05.721357 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.721341 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:05.721357 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.721358 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:05.721466 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.721370 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:05.721466 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:05.721420 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:07.721403118 +0000 UTC m=+6.230555560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:05.930730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.930614 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:29:03 +0000 UTC" deadline="2028-02-25 01:45:08.053937605 +0000 UTC" Sep 08 18:34:05.930730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:05.930685 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12823h11m2.123256507s" Sep 08 18:34:06.021842 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.021815 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:06.021993 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:06.021940 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:06.022357 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.022337 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:06.022467 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:06.022445 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:06.022544 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.022513 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:06.022611 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:06.022594 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:06.048951 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.048921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqll6" event={"ID":"4766c049-9d22-4ff0-865c-d7935eddcda9","Type":"ContainerStarted","Data":"0f6f02d2c973d02f75046f927bf384d37051c7ec368abcbac7166ddbc2b5eafc"} Sep 08 18:34:06.052409 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.052339 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkdtb" event={"ID":"c65b4c43-1caf-4823-adc2-653bb8ff7a08","Type":"ContainerStarted","Data":"868937b8a2d6a68d7350d69106866bd1b800e745c8e1a932c0dccfcce5839426"} Sep 08 18:34:06.055675 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.055599 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" event={"ID":"5a3853aaaa6ace0991cc2a038a9653ab","Type":"ContainerStarted","Data":"e7a56c58265c8f87bf147f7fbc6624625882382144f6d33ad4d37a74db792afe"} Sep 08 18:34:06.069995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.069436 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-251.ec2.internal" podStartSLOduration=3.069420437 podStartE2EDuration="3.069420437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:06.068635547 +0000 UTC m=+4.577787989" watchObservedRunningTime="2026-09-08 18:34:06.069420437 +0000 UTC m=+4.578572879" Sep 08 18:34:06.070290 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.070266 2557 generic.go:356] "Generic (PLEG): container finished" podID="14f96a47f3d21417de7f2c2d7e4702ad" containerID="db5428d5a5037736db3172a09c569db1260cde69611e208211c44834bd0f9372" exitCode=0 Sep 08 18:34:06.070376 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.070343 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" event={"ID":"14f96a47f3d21417de7f2c2d7e4702ad","Type":"ContainerDied","Data":"db5428d5a5037736db3172a09c569db1260cde69611e208211c44834bd0f9372"} Sep 08 18:34:06.073607 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:06.073554 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" event={"ID":"34678d45-07b6-4bed-88dd-a083983489f3","Type":"ContainerStarted","Data":"60189109b0f69ba3d874964152882c616ca8385434542e79373e19e176ff401e"} Sep 08 18:34:07.091289 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:07.091227 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" event={"ID":"14f96a47f3d21417de7f2c2d7e4702ad","Type":"ContainerStarted","Data":"8ac47d6ff333e2c8c8626f106bcb4a3f7009aaed0ab21b65ff2d431a11bfe832"} Sep 08 18:34:07.637194 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:07.637034 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:07.637194 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:07.637104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:07.637425 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.637235 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:07.637425 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.637291 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:11.637273766 +0000 UTC m=+10.146426191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:07.637699 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.637678 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:07.637797 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.637735 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:34:11.637718503 +0000 UTC m=+10.146870969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:07.738176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:07.738138 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:07.738327 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.738296 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:07.738327 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.738315 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:07.738327 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.738327 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:07.738488 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:07.738376 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:11.738360224 +0000 UTC m=+10.247512658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:08.019560 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:08.018966 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:08.019560 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:08.019077 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:08.019560 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:08.019426 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:08.019560 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:08.019518 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:08.020200 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:08.020015 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:08.020200 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:08.020122 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:10.019186 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:10.019149 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:10.019660 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:10.019150 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:10.019660 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:10.019287 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:10.019660 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:10.019298 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:10.019660 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:10.019365 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:10.019660 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:10.019450 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:11.674769 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:11.674714 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:11.674769 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:11.674787 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:11.675445 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.674860 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:11.675445 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.674924 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:11.675445 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.674930 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:34:19.674908421 +0000 UTC m=+18.184060857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:11.675445 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.674986 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:19.674969452 +0000 UTC m=+18.184121876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:11.775602 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:11.775515 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:11.775777 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.775710 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:11.775777 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.775733 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:11.775777 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.775746 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:11.775920 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:11.775802 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:19.775784906 +0000 UTC m=+18.284937342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:12.019877 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:12.019852 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:12.020053 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:12.019890 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:12.020053 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:12.019966 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:12.020296 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:12.020210 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:12.020385 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:12.020324 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:12.020493 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:12.020445 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:14.018850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:14.018815 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:14.018850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:14.018856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:14.019293 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:14.018827 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:14.019293 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:14.018941 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:14.019293 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:14.019067 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:14.019293 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:14.019124 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:16.019112 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:16.019080 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:16.019577 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:16.019087 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:16.019577 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:16.019209 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:16.019577 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:16.019221 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:16.019577 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:16.019307 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:16.019577 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:16.019408 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:18.019000 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:18.018969 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:18.019000 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:18.019000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:18.019501 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:18.018973 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:18.019501 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:18.019148 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:18.019501 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:18.019224 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:18.019501 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:18.019316 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:19.732389 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:19.732353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:19.732880 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:19.732512 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:19.732880 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.732513 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:19.732880 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.732603 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:35.732583279 +0000 UTC m=+34.241735703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:19.732880 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.732627 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:19.732880 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.732700 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:34:35.732682634 +0000 UTC m=+34.241835070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:19.832916 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:19.832890 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:19.833073 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.833012 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:19.833073 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.833032 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:19.833073 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.833045 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:19.833183 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:19.833103 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:35.833087225 +0000 UTC m=+34.342239645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:20.018767 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:20.018685 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:20.018767 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:20.018709 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:20.018767 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:20.018693 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:20.019008 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:20.018807 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:20.019008 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:20.018879 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:20.019008 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:20.018966 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:22.022353 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:22.022319 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:22.022353 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:22.022343 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:22.022893 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:22.022452 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:22.022893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:22.022463 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:22.022893 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:22.022594 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:22.022893 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:22.022703 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:23.124609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.124253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" event={"ID":"34678d45-07b6-4bed-88dd-a083983489f3","Type":"ContainerStarted","Data":"dc21bad6b595c9ce245b3325cfec24baca1ee1b6b6cbb2fb561ea7e2405e8513"} Sep 08 18:34:23.125884 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.125859 2557 generic.go:356] "Generic (PLEG): container finished" podID="2547b1f7-3386-4ba6-ba73-e0efcdfbb57b" containerID="5c43e1de492ea574c28e727f1fb96a64c619b81ead1206f9e780847ee7439adb" exitCode=0 Sep 08 18:34:23.125984 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.125939 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerDied","Data":"5c43e1de492ea574c28e727f1fb96a64c619b81ead1206f9e780847ee7439adb"} Sep 08 18:34:23.128425 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.128402 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"18677feb7a0df1cf27e279a6a3a8e05b9280d0c1850b5bf2cf0cdb6894f28429"} Sep 08 18:34:23.128514 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.128473 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"0a9e3465aaa2a84013645d69e335e5b465f2d070f83a36aa6c8e6bb51f64029d"} Sep 08 18:34:23.128579 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.128521 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"a4a90e2e5b8059a2d9000adfc03f0c41ba4341f94552b9f91d0c790a43405585"} Sep 08 18:34:23.129839 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.129818 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5m7lw" event={"ID":"728eba11-c5b4-4f60-afed-b980d81a15e6","Type":"ContainerStarted","Data":"fceed868562a6ca59bac83e24cc734b7886b25e8dc8ae23aa47992294ae87658"} Sep 08 18:34:23.131566 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.131539 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqll6" event={"ID":"4766c049-9d22-4ff0-865c-d7935eddcda9","Type":"ContainerStarted","Data":"63460a81708524a077df908708caaca32b95cfac2fc4c82ed073064a4fa8e86b"} Sep 08 18:34:23.133695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.133612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkdtb" event={"ID":"c65b4c43-1caf-4823-adc2-653bb8ff7a08","Type":"ContainerStarted","Data":"14accf76467235fa61be90a51744e9ea47d063197de6e6db4494daa9d60c099e"} Sep 08 18:34:23.135360 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.135336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" event={"ID":"340d0037-0c5e-4280-b788-68537331aa79","Type":"ContainerStarted","Data":"e534e156198a8eff317dafdae66b45d58b49faf4042da96842b325da99aa0704"} Sep 08 18:34:23.136703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.136681 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ntg6k" event={"ID":"2e030b87-5747-4a52-ae8a-1f33fe710f9b","Type":"ContainerStarted","Data":"06ff8e6df4ba38e41094ff93cb535588a0018f2cf6e0d494b1f02213d0ab1a7a"} Sep 08 18:34:23.140606 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.140568 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j9n4r" podStartSLOduration=3.525433637 podStartE2EDuration="21.140556733s" totalImagesPullingTime="17.615123096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.140088382 +0000 UTC m=+21.649240824" watchObservedRunningTime="2026-09-08 18:34:23.140556733 +0000 UTC m=+21.649709244" Sep 08 18:34:23.140736 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.140707 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-251.ec2.internal" podStartSLOduration=20.140696007 podStartE2EDuration="20.140696007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:07.106488991 +0000 UTC m=+5.615641435" watchObservedRunningTime="2026-09-08 18:34:23.140696007 +0000 UTC m=+21.649848447" Sep 08 18:34:23.153051 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.152981 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mqll6" podStartSLOduration=10.896559494 podStartE2EDuration="21.15297117s" totalImagesPullingTime="10.256411676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.152814095 +0000 UTC m=+21.661966536" watchObservedRunningTime="2026-09-08 18:34:23.15297117 +0000 UTC m=+21.662123611" Sep 08 18:34:23.165418 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.165386 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ntg6k" podStartSLOduration=3.537847419 podStartE2EDuration="21.165375687s" totalImagesPullingTime="17.627528268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.165014752 +0000 UTC m=+21.674167193" watchObservedRunningTime="2026-09-08 18:34:23.165375687 +0000 UTC m=+21.674528128" Sep 08 18:34:23.196254 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.196222 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5m7lw" podStartSLOduration=3.567245271 podStartE2EDuration="21.196211749s" totalImagesPullingTime="17.628966478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.177090343 +0000 UTC m=+21.686242783" watchObservedRunningTime="2026-09-08 18:34:23.196211749 +0000 UTC m=+21.705364189" Sep 08 18:34:23.196338 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.196281 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dkdtb" podStartSLOduration=3.374324249 podStartE2EDuration="21.196276983s" totalImagesPullingTime="17.821952734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.195943329 +0000 UTC m=+21.705095771" watchObservedRunningTime="2026-09-08 18:34:23.196276983 +0000 UTC m=+21.705429446" Sep 08 18:34:23.963021 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:23.962971 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 18:34:24.019713 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.019687 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:24.019841 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.019794 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:24.019841 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:24.019801 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:24.019954 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:24.019915 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:24.019954 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.019940 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:24.020058 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:24.019992 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:24.140441 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.140377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" event={"ID":"340d0037-0c5e-4280-b788-68537331aa79","Type":"ContainerStarted","Data":"9031c85cc4db2c21d3e54d441f30d08f2099218b0fed6e1af476ae00bcff788e"} Sep 08 18:34:24.142064 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.142030 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvmjr" event={"ID":"e8b0b394-f387-4007-8ef3-46024a45b449","Type":"ContainerStarted","Data":"fb59fbe931c1df2f407872c03624f20a75a99d37cdcd1a70c701f5b3f2c87bf2"} Sep 08 18:34:24.146148 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.146122 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"0d7929004fbb5824184187688d88f7b00a4bfab27762fdab645687b16415c0fe"} Sep 08 18:34:24.146262 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.146200 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"1cdd6e9a76f37625eda159da79356eaa8a6ab1368bbc79cdc8e0350437d68617"} Sep 08 18:34:24.146262 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.146249 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"d29d4e9d1f5b28cd1fcd343291a3a42ccb1b396a2da26f40830a19c034c8e43a"} Sep 08 18:34:24.158426 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.158369 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pvmjr" podStartSLOduration=4.533312385 podStartE2EDuration="22.158353122s" totalImagesPullingTime="17.625040737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:24.15808671 +0000 UTC m=+22.667239151" watchObservedRunningTime="2026-09-08 18:34:24.158353122 +0000 UTC m=+22.667505565" Sep 08 18:34:24.948070 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.947982 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T18:34:23.963030433Z","UUID":"07413885-920f-4b1c-949e-ed00b85e185c","Handler":null,"Name":"","Endpoint":""} Sep 08 18:34:24.949633 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.949607 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 18:34:24.949633 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:24.949655 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 18:34:26.019605 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.019576 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:26.019605 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.019589 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:26.020125 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:26.019711 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:26.020125 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:26.019800 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:26.020125 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.019856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:26.020125 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:26.019932 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:26.153675 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.153626 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"ae5391fcb7e76f4eec639e775add9ced072f2ab105b001506cc24b7d53f5b7dc"} Sep 08 18:34:26.155984 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.155957 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" event={"ID":"340d0037-0c5e-4280-b788-68537331aa79","Type":"ContainerStarted","Data":"9841f1488ee859457fc9a36cc6929b4801a7aaa47163af8d18ed7b49220c3e24"} Sep 08 18:34:26.174311 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.174242 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvws5" podStartSLOduration=4.480227154 podStartE2EDuration="24.174228943s" totalImagesPullingTime="19.694001789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:26.173483774 +0000 UTC m=+24.682636216" watchObservedRunningTime="2026-09-08 18:34:26.174228943 +0000 UTC m=+24.683381383" Sep 08 18:34:26.715842 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:26.715808 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:27.090672 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:27.090621 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:27.091216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:27.091196 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:27.158514 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:27.158491 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mqll6" Sep 08 18:34:28.019868 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.019717 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:28.020017 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.019716 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:28.020017 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:28.019933 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:28.020118 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:28.020022 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:28.020118 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.019716 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:28.020118 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:28.020089 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:28.160928 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.160906 2557 generic.go:356] "Generic (PLEG): container finished" podID="2547b1f7-3386-4ba6-ba73-e0efcdfbb57b" containerID="f720fad631cdbd2e0cddb6bad9e8424e8b75573d50c5853969b3da2749dbb31c" exitCode=0 Sep 08 18:34:28.161617 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.160973 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerDied","Data":"f720fad631cdbd2e0cddb6bad9e8424e8b75573d50c5853969b3da2749dbb31c"} Sep 08 18:34:28.168090 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.168061 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" event={"ID":"61de7352-8afc-4cb4-b04e-9b1556b7bd6a","Type":"ContainerStarted","Data":"dee5f1fea995aae84c6cc03e415cc6b73c32d872b00d681621e2876d58ca1044"} Sep 08 18:34:28.168238 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.168218 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:28.205890 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:28.205849 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" podStartSLOduration=8.545999315 podStartE2EDuration="26.205838153s" totalImagesPullingTime="17.659838838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:28.205360153 +0000 UTC m=+26.714512593" watchObservedRunningTime="2026-09-08 18:34:28.205838153 +0000 UTC m=+26.714990593" Sep 08 18:34:29.171804 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.171694 2557 generic.go:356] "Generic (PLEG): container finished" podID="2547b1f7-3386-4ba6-ba73-e0efcdfbb57b" containerID="bf6232ecc1fad96f7327602ed22c25c76c1b6d63111da2b2037af4588ee14f6e" exitCode=0 Sep 08 18:34:29.171804 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.171780 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerDied","Data":"bf6232ecc1fad96f7327602ed22c25c76c1b6d63111da2b2037af4588ee14f6e"} Sep 08 18:34:29.172457 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.172441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:29.172514 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.172506 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:29.187075 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.187054 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:29.187279 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.187256 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:34:29.492411 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.492387 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75t6m"] Sep 08 18:34:29.492597 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.492489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:29.492597 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:29.492572 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:29.495266 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.495215 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4c29z"] Sep 08 18:34:29.495370 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.495354 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:29.495483 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:29.495461 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:29.495887 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.495867 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7rdgz"] Sep 08 18:34:29.496004 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:29.495988 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:29.496105 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:29.496083 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:30.176126 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:30.176104 2557 generic.go:356] "Generic (PLEG): container finished" podID="2547b1f7-3386-4ba6-ba73-e0efcdfbb57b" containerID="8d38606b359f89c7505e70f19f1cd188c10949813fac7a471b7ff2341ab2f873" exitCode=0 Sep 08 18:34:30.176393 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:30.176187 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerDied","Data":"8d38606b359f89c7505e70f19f1cd188c10949813fac7a471b7ff2341ab2f873"} Sep 08 18:34:31.019283 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:31.019252 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:31.019429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:31.019349 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:31.019429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:31.019361 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:31.019429 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:31.019360 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:31.019548 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:31.019431 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:31.019548 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:31.019517 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:33.019050 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:33.018870 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:33.019381 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:33.018915 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:33.019381 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:33.019116 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:33.019381 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:33.018927 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:33.019381 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:33.019192 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:33.019381 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:33.019280 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:35.019212 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.019182 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:35.019212 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.019197 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:35.019758 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.019292 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75t6m" podUID="06143e3a-eb52-4a46-bf0d-9758bfff5e50" Sep 08 18:34:35.019758 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.019374 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4c29z" podUID="2a9a9061-40d6-41b5-a0f5-f5f0d686ba22" Sep 08 18:34:35.019758 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.019443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:35.019758 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.019525 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:34:35.748404 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.748376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:35.748498 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.748453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:35.748552 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.748511 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:35.748552 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.748545 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:35.748631 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.748592 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret podName:2a9a9061-40d6-41b5-a0f5-f5f0d686ba22 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:07.748571202 +0000 UTC m=+66.257723635 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret") pod "global-pull-secret-syncer-4c29z" (UID: "2a9a9061-40d6-41b5-a0f5-f5f0d686ba22") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:35.748631 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.748613 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:35:07.748602467 +0000 UTC m=+66.257754886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:35.748915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.748898 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-251.ec2.internal" event="NodeReady" Sep 08 18:34:35.748984 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.748973 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 18:34:35.790695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.790665 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-txkkw"] Sep 08 18:34:35.821205 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.821181 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2klms"] Sep 08 18:34:35.821306 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.821292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:35.823302 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.823283 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 18:34:35.823604 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.823590 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rsh8d\"" Sep 08 18:34:35.823818 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.823802 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 18:34:35.835352 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.835331 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2klms"] Sep 08 18:34:35.835446 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.835363 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-txkkw"] Sep 08 18:34:35.835566 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.835460 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:35.837527 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.837506 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 18:34:35.837686 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.837665 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 18:34:35.837799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.837777 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 18:34:35.837922 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.837805 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9b5jk\"" Sep 08 18:34:35.848756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.848730 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:35.848898 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.848866 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:35.848898 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.848885 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:35.848898 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.848895 2557 projected.go:196] Error preparing data for projected volume kube-api-access-z6lxx for pod openshift-network-diagnostics/network-check-target-75t6m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:35.849100 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:35.848977 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx podName:06143e3a-eb52-4a46-bf0d-9758bfff5e50 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:07.848937987 +0000 UTC m=+66.358090410 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6lxx" (UniqueName: "kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx") pod "network-check-target-75t6m" (UID: "06143e3a-eb52-4a46-bf0d-9758bfff5e50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:35.949939 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.949916 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvvp\" (UniqueName: \"kubernetes.io/projected/df8f149c-de77-4443-ab61-d9d549496c9a-kube-api-access-7xvvp\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:35.950038 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.949946 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d730a5-89d1-4b08-8e23-df1ab8568b99-cert\") pod \"ingress-canary-2klms\" (UID: \"97d730a5-89d1-4b08-8e23-df1ab8568b99\") " pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:35.950038 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.949969 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqvv\" (UniqueName: \"kubernetes.io/projected/97d730a5-89d1-4b08-8e23-df1ab8568b99-kube-api-access-ptqvv\") pod \"ingress-canary-2klms\" (UID: \"97d730a5-89d1-4b08-8e23-df1ab8568b99\") " pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:35.950144 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.950073 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8f149c-de77-4443-ab61-d9d549496c9a-tmp-dir\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:35.950144 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.950125 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df8f149c-de77-4443-ab61-d9d549496c9a-config-volume\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:35.950238 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:35.950165 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.051157 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051129 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvvp\" (UniqueName: \"kubernetes.io/projected/df8f149c-de77-4443-ab61-d9d549496c9a-kube-api-access-7xvvp\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.051157 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d730a5-89d1-4b08-8e23-df1ab8568b99-cert\") pod \"ingress-canary-2klms\" (UID: \"97d730a5-89d1-4b08-8e23-df1ab8568b99\") " pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:36.051589 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqvv\" (UniqueName: \"kubernetes.io/projected/97d730a5-89d1-4b08-8e23-df1ab8568b99-kube-api-access-ptqvv\") pod \"ingress-canary-2klms\" (UID: \"97d730a5-89d1-4b08-8e23-df1ab8568b99\") " pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:36.051589 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8f149c-de77-4443-ab61-d9d549496c9a-tmp-dir\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.051589 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051390 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df8f149c-de77-4443-ab61-d9d549496c9a-config-volume\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.051589 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051427 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.051589 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:36.051548 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:36.051797 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:36.051624 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:34:36.551604785 +0000 UTC m=+35.060757214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:34:36.051797 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051696 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8f149c-de77-4443-ab61-d9d549496c9a-tmp-dir\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.051915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.051898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df8f149c-de77-4443-ab61-d9d549496c9a-config-volume\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.055012 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.054993 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d730a5-89d1-4b08-8e23-df1ab8568b99-cert\") pod \"ingress-canary-2klms\" (UID: \"97d730a5-89d1-4b08-8e23-df1ab8568b99\") " pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:36.059221 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.059201 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvvp\" (UniqueName: \"kubernetes.io/projected/df8f149c-de77-4443-ab61-d9d549496c9a-kube-api-access-7xvvp\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.059553 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.059535 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqvv\" (UniqueName: \"kubernetes.io/projected/97d730a5-89d1-4b08-8e23-df1ab8568b99-kube-api-access-ptqvv\") pod \"ingress-canary-2klms\" (UID: \"97d730a5-89d1-4b08-8e23-df1ab8568b99\") " pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:36.143639 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.143608 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2klms" Sep 08 18:34:36.194349 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.194272 2557 generic.go:356] "Generic (PLEG): container finished" podID="2547b1f7-3386-4ba6-ba73-e0efcdfbb57b" containerID="e6f0cda798d67f0c07a1cacf54e0c634bdb598b741faa246cc422ad2c87243dd" exitCode=0 Sep 08 18:34:36.194460 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.194351 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerDied","Data":"e6f0cda798d67f0c07a1cacf54e0c634bdb598b741faa246cc422ad2c87243dd"} Sep 08 18:34:36.444606 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.444538 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2klms"] Sep 08 18:34:36.447695 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:34:36.447671 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d730a5_89d1_4b08_8e23_df1ab8568b99.slice/crio-ff3cf7b740b910b1a2ef81bb0f702f64678fed69c8c0fb01d94c6b6c5b1a0b6c WatchSource:0}: Error finding container ff3cf7b740b910b1a2ef81bb0f702f64678fed69c8c0fb01d94c6b6c5b1a0b6c: Status 404 returned error can't find the container with id ff3cf7b740b910b1a2ef81bb0f702f64678fed69c8c0fb01d94c6b6c5b1a0b6c Sep 08 18:34:36.555080 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:36.555056 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:36.555177 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:36.555165 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:36.555215 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:36.555210 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:34:37.555197989 +0000 UTC m=+36.064350407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:34:37.019149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.019104 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:34:37.019313 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.019219 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:34:37.019400 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.019373 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:34:37.021602 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.021584 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 18:34:37.021692 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.021627 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 18:34:37.021758 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.021585 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z79bg\"" Sep 08 18:34:37.021979 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.021952 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 18:34:37.022173 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.022160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 18:34:37.022285 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.022259 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmn47\"" Sep 08 18:34:37.198611 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.198576 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2klms" event={"ID":"97d730a5-89d1-4b08-8e23-df1ab8568b99","Type":"ContainerStarted","Data":"ff3cf7b740b910b1a2ef81bb0f702f64678fed69c8c0fb01d94c6b6c5b1a0b6c"} Sep 08 18:34:37.202240 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.202217 2557 generic.go:356] "Generic (PLEG): container finished" podID="2547b1f7-3386-4ba6-ba73-e0efcdfbb57b" containerID="ba41e85182e0fe0f26c830e677d043ac789063d578510ff4fa3875983a3fae4e" exitCode=0 Sep 08 18:34:37.202354 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.202252 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerDied","Data":"ba41e85182e0fe0f26c830e677d043ac789063d578510ff4fa3875983a3fae4e"} Sep 08 18:34:37.561891 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:37.561799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:37.562047 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:37.561955 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:37.562047 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:37.562020 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:34:39.561999544 +0000 UTC m=+38.071151962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:34:38.208975 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:38.208945 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" event={"ID":"2547b1f7-3386-4ba6-ba73-e0efcdfbb57b","Type":"ContainerStarted","Data":"0cd07ceee4691941f99f977314d71307b88614485962e85bc217f376f3bc99b1"} Sep 08 18:34:38.228033 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:38.227982 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z7nvh" podStartSLOduration=7.242013216 podStartE2EDuration="36.227966381s" totalImagesPullingTime="28.985953165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:38.227932799 +0000 UTC m=+36.737085252" watchObservedRunningTime="2026-09-08 18:34:38.227966381 +0000 UTC m=+36.737118803" Sep 08 18:34:39.212940 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:39.212908 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2klms" event={"ID":"97d730a5-89d1-4b08-8e23-df1ab8568b99","Type":"ContainerStarted","Data":"0974411d122e6a019f2b9a92fd83606d39d9d92f132c108af747102e9be56c58"} Sep 08 18:34:39.229283 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:39.229239 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2klms" podStartSLOduration=1.9364277300000001 podStartE2EDuration="4.229224334s" totalImagesPullingTime="2.292796604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:39.228933834 +0000 UTC m=+37.738086276" watchObservedRunningTime="2026-09-08 18:34:39.229224334 +0000 UTC m=+37.738376769" Sep 08 18:34:39.576637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:39.576606 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:39.576812 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:39.576774 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:39.576874 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:39.576847 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:34:43.576826795 +0000 UTC m=+42.085979227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:34:43.603466 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:43.603301 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:43.603466 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:43.603461 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:43.603877 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:43.603523 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:34:51.603504142 +0000 UTC m=+50.112656562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:34:51.650481 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:34:51.650443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:34:51.650924 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:51.650588 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:51.650924 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:34:51.650670 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:35:07.65063419 +0000 UTC m=+66.159786609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:35:01.188482 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:01.188452 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttvkc" Sep 08 18:35:07.750527 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.750487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:35:07.750527 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.750539 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:35:07.751053 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.750559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:35:07.751053 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:35:07.750628 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:07.751053 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:35:07.750698 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:35:39.750683058 +0000 UTC m=+98.259835481 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:35:07.752818 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.752803 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 18:35:07.752870 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.752809 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 18:35:07.761181 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:35:07.761164 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 18:35:07.761309 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:35:07.761221 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:36:11.761204524 +0000 UTC m=+130.270356964 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : secret "metrics-daemon-secret" not found Sep 08 18:35:07.764541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.764521 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a9a9061-40d6-41b5-a0f5-f5f0d686ba22-original-pull-secret\") pod \"global-pull-secret-syncer-4c29z\" (UID: \"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22\") " pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:35:07.851670 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.851634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:35:07.853688 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.853674 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 18:35:07.863791 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.863777 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 18:35:07.874363 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.874339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6lxx\" (UniqueName: \"kubernetes.io/projected/06143e3a-eb52-4a46-bf0d-9758bfff5e50-kube-api-access-z6lxx\") pod \"network-check-target-75t6m\" (UID: \"06143e3a-eb52-4a46-bf0d-9758bfff5e50\") " pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:35:07.931995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.931975 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmn47\"" Sep 08 18:35:07.938635 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.938619 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4c29z" Sep 08 18:35:07.940385 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:07.940372 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:35:08.206955 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:08.206926 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4c29z"] Sep 08 18:35:08.210464 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:08.210432 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75t6m"] Sep 08 18:35:08.210881 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:35:08.210853 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9a9061_40d6_41b5_a0f5_f5f0d686ba22.slice/crio-99211c4b22798bf83c01c445519f2eefac7f11e0726ce712541479df662d269f WatchSource:0}: Error finding container 99211c4b22798bf83c01c445519f2eefac7f11e0726ce712541479df662d269f: Status 404 returned error can't find the container with id 99211c4b22798bf83c01c445519f2eefac7f11e0726ce712541479df662d269f Sep 08 18:35:08.213869 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:35:08.213820 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06143e3a_eb52_4a46_bf0d_9758bfff5e50.slice/crio-7b1bd194ec96337e5c0b41f60392f6b3d9d260e63a0a414c5d0cc784785542e8 WatchSource:0}: Error finding container 7b1bd194ec96337e5c0b41f60392f6b3d9d260e63a0a414c5d0cc784785542e8: Status 404 returned error can't find the container with id 7b1bd194ec96337e5c0b41f60392f6b3d9d260e63a0a414c5d0cc784785542e8 Sep 08 18:35:08.273825 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:08.273773 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75t6m" event={"ID":"06143e3a-eb52-4a46-bf0d-9758bfff5e50","Type":"ContainerStarted","Data":"7b1bd194ec96337e5c0b41f60392f6b3d9d260e63a0a414c5d0cc784785542e8"} Sep 08 18:35:08.274802 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:08.274784 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4c29z" event={"ID":"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22","Type":"ContainerStarted","Data":"99211c4b22798bf83c01c445519f2eefac7f11e0726ce712541479df662d269f"} Sep 08 18:35:13.288386 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:13.288352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75t6m" event={"ID":"06143e3a-eb52-4a46-bf0d-9758bfff5e50","Type":"ContainerStarted","Data":"112a767951ee52af4a11fd8418c97c57e54955827972906fe38d6051c1e0b8b2"} Sep 08 18:35:13.288914 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:13.288473 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:35:13.289943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:13.289920 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4c29z" event={"ID":"2a9a9061-40d6-41b5-a0f5-f5f0d686ba22","Type":"ContainerStarted","Data":"5c9f12b07c1799d26fe0f1e7e1c415c959b30de9676b0949530b90b0995ba451"} Sep 08 18:35:13.301596 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:13.301557 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-75t6m" podStartSLOduration=66.608982517 podStartE2EDuration="1m11.301545924s" totalImagesPullingTime="4.692563407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:13.300916887 +0000 UTC m=+71.810069330" watchObservedRunningTime="2026-09-08 18:35:13.301545924 +0000 UTC m=+71.810698361" Sep 08 18:35:39.850153 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:39.850117 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:35:39.850525 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:35:39.850273 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:39.850525 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:35:39.850347 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:36:43.850330894 +0000 UTC m=+162.359483313 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:35:44.294777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:44.294747 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-75t6m" Sep 08 18:35:44.309225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:44.309184 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4c29z" podStartSLOduration=97.612704393 podStartE2EDuration="1m42.309170324s" totalImagesPullingTime="4.696465931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:13.313042718 +0000 UTC m=+71.822195158" watchObservedRunningTime="2026-09-08 18:35:44.309170324 +0000 UTC m=+102.818322765" Sep 08 18:35:55.980365 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.980333 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv"] Sep 08 18:35:55.982016 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.981994 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:55.984236 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.984208 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 08 18:35:55.984370 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.984248 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 18:35:55.984673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.984637 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 08 18:35:55.984771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.984672 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 08 18:35:55.985106 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.985089 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 18:35:55.985405 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.985391 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 18:35:55.985453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.985392 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 08 18:35:55.993145 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:55.993123 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv"] Sep 08 18:35:56.048064 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.048042 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-hub\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.048149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.048087 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.048149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.048122 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.048219 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.048176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-ca\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.048252 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.048214 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgd4k\" (UniqueName: \"kubernetes.io/projected/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-kube-api-access-rgd4k\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.048324 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.048280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.149521 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.149501 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.149595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.149529 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-hub\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.149595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.149547 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.149595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.149571 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.149755 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.149596 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-ca\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.149755 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.149624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd4k\" (UniqueName: \"kubernetes.io/projected/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-kube-api-access-rgd4k\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.150375 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.150352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.151978 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.151952 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.152085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.151985 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-hub\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.152085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.151995 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-ca\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.155045 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.155025 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.157311 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.157292 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd4k\" (UniqueName: \"kubernetes.io/projected/d8e0ebd8-3620-4114-b4e6-b43077c9c6c3-kube-api-access-rgd4k\") pod \"cluster-proxy-proxy-agent-57f968865-vlcbv\" (UID: \"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.305117 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.305073 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:35:56.569532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:56.569460 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv"] Sep 08 18:35:56.573436 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:35:56.573410 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e0ebd8_3620_4114_b4e6_b43077c9c6c3.slice/crio-357a00f6bcfd14890950d5b041c4a08016948464ffccfbace2861b0b9be5c7b6 WatchSource:0}: Error finding container 357a00f6bcfd14890950d5b041c4a08016948464ffccfbace2861b0b9be5c7b6: Status 404 returned error can't find the container with id 357a00f6bcfd14890950d5b041c4a08016948464ffccfbace2861b0b9be5c7b6 Sep 08 18:35:57.389962 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:35:57.389929 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" event={"ID":"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3","Type":"ContainerStarted","Data":"357a00f6bcfd14890950d5b041c4a08016948464ffccfbace2861b0b9be5c7b6"} Sep 08 18:36:01.403795 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:01.403732 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" event={"ID":"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3","Type":"ContainerStarted","Data":"b9a6397d9bd75605ca9f2beba19ec49cde361b2f4a51b0f514c78ccde142e9ca"} Sep 08 18:36:01.403795 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:01.403776 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" event={"ID":"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3","Type":"ContainerStarted","Data":"b42c7128b2990ec5928f274cd5b8acc95ad785279226a5253d0e67028a90107c"} Sep 08 18:36:01.403795 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:01.403790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" event={"ID":"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3","Type":"ContainerStarted","Data":"bdc9f316a1ee9725615944df1de8d4730bbbaeb664bafc2c62357eb730d01fb5"} Sep 08 18:36:01.419680 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:01.419621 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" podStartSLOduration=1.9688694469999999 podStartE2EDuration="6.419607798s" totalImagesPullingTime="4.450738351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:01.419161139 +0000 UTC m=+119.928313579" watchObservedRunningTime="2026-09-08 18:36:01.419607798 +0000 UTC m=+119.928760238" Sep 08 18:36:11.851115 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:11.851074 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:36:11.851494 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:36:11.851188 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 18:36:11.851494 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:36:11.851265 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:38:13.851251159 +0000 UTC m=+252.360403583 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : secret "metrics-daemon-secret" not found Sep 08 18:36:38.830010 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:36:38.829959 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-txkkw" podUID="df8f149c-de77-4443-ab61-d9d549496c9a" Sep 08 18:36:39.501468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:39.501442 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-txkkw" Sep 08 18:36:40.043610 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:36:40.043577 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:36:43.853050 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:43.853006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:36:43.853531 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:36:43.853173 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:36:43.853531 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:36:43.853251 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls podName:df8f149c-de77-4443-ab61-d9d549496c9a nodeName:}" failed. No retries permitted until 2026-09-08 18:38:45.853230126 +0000 UTC m=+284.362382545 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls") pod "dns-default-txkkw" (UID: "df8f149c-de77-4443-ab61-d9d549496c9a") : secret "dns-default-metrics-tls" not found Sep 08 18:36:51.019359 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:36:51.019325 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:38:13.925724 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:13.925685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:38:13.926184 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:13.925799 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 18:38:13.926184 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:13.925860 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs podName:04151103-1947-499e-a096-b70859a454ff nodeName:}" failed. No retries permitted until 2026-09-08 18:40:15.925846023 +0000 UTC m=+374.434998443 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs") pod "network-metrics-daemon-7rdgz" (UID: "04151103-1947-499e-a096-b70859a454ff") : secret "metrics-daemon-secret" not found Sep 08 18:38:26.264987 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.264914 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b"] Sep 08 18:38:26.267743 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.267721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.269914 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.269898 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 08 18:38:26.270432 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.270414 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 08 18:38:26.270908 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.270890 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pvznm\"" Sep 08 18:38:26.270966 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.270890 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 08 18:38:26.271183 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.271169 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:38:26.278058 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.278037 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b"] Sep 08 18:38:26.302480 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.302454 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b459ecd6-967b-4efe-9567-c782a158e362-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.302572 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.302494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b459ecd6-967b-4efe-9567-c782a158e362-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.302572 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.302519 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhr99\" (UniqueName: \"kubernetes.io/projected/b459ecd6-967b-4efe-9567-c782a158e362-kube-api-access-vhr99\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.402884 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.402862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b459ecd6-967b-4efe-9567-c782a158e362-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.402964 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.402891 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhr99\" (UniqueName: \"kubernetes.io/projected/b459ecd6-967b-4efe-9567-c782a158e362-kube-api-access-vhr99\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.402964 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.402946 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b459ecd6-967b-4efe-9567-c782a158e362-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.403307 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.403291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b459ecd6-967b-4efe-9567-c782a158e362-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.404976 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.404960 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b459ecd6-967b-4efe-9567-c782a158e362-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.414048 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.414022 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhr99\" (UniqueName: \"kubernetes.io/projected/b459ecd6-967b-4efe-9567-c782a158e362-kube-api-access-vhr99\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hdx4b\" (UID: \"b459ecd6-967b-4efe-9567-c782a158e362\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.576524 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.576475 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" Sep 08 18:38:26.843303 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:26.843239 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b"] Sep 08 18:38:26.846290 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:38:26.846265 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb459ecd6_967b_4efe_9567_c782a158e362.slice/crio-c9f65dbeeb7a528da7535a01d0382f7f538a5081947265109b9ef5fdeb45ee81 WatchSource:0}: Error finding container c9f65dbeeb7a528da7535a01d0382f7f538a5081947265109b9ef5fdeb45ee81: Status 404 returned error can't find the container with id c9f65dbeeb7a528da7535a01d0382f7f538a5081947265109b9ef5fdeb45ee81 Sep 08 18:38:27.786550 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:27.786510 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" event={"ID":"b459ecd6-967b-4efe-9567-c782a158e362","Type":"ContainerStarted","Data":"c9f65dbeeb7a528da7535a01d0382f7f538a5081947265109b9ef5fdeb45ee81"} Sep 08 18:38:29.794030 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:29.794000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" event={"ID":"b459ecd6-967b-4efe-9567-c782a158e362","Type":"ContainerStarted","Data":"fcd5755b9d3efd3fbedc5ea7626fb068d92c744c801c333b4303f0d6276d8593"} Sep 08 18:38:29.807714 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:29.807664 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" podStartSLOduration=1.5470379730000001 podStartE2EDuration="3.807623691s" totalImagesPullingTime="2.260585718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:29.807404524 +0000 UTC m=+268.316556965" watchObservedRunningTime="2026-09-08 18:38:29.807623691 +0000 UTC m=+268.316776132" Sep 08 18:38:30.944693 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.944664 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9c4dq"] Sep 08 18:38:30.947632 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.947617 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:30.949923 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.949902 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 18:38:30.950513 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.950498 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 18:38:30.950566 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.950525 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 18:38:30.950566 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.950530 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q24t4\"" Sep 08 18:38:30.950672 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.950525 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 18:38:30.959099 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:30.959080 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9c4dq"] Sep 08 18:38:31.037471 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.037446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.037576 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.037472 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.037576 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.037493 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbzw\" (UniqueName: \"kubernetes.io/projected/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-api-access-wjbzw\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.037576 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.037510 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-crio-socket\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.037576 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.037560 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-data-volume\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138519 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138439 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138519 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138470 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbzw\" (UniqueName: \"kubernetes.io/projected/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-api-access-wjbzw\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138519 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138509 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-crio-socket\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138691 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138534 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-data-volume\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138691 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.138533 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:31.138691 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-crio-socket\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.138691 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.138609 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:31.638592543 +0000 UTC m=+270.147744967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : configmap "kube-rbac-proxy" not found Sep 08 18:38:31.138691 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.138633 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.138947 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.138716 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:31.638700388 +0000 UTC m=+270.147852824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.138947 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.138919 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-data-volume\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.146480 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.146460 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbzw\" (UniqueName: \"kubernetes.io/projected/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-api-access-wjbzw\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.642145 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.642115 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.642373 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:31.642149 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:31.642373 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.642238 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:31.642373 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.642256 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.642373 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.642312 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:32.642294277 +0000 UTC m=+271.151446696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : configmap "kube-rbac-proxy" not found Sep 08 18:38:31.642373 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:31.642331 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:32.642321235 +0000 UTC m=+271.151473654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:32.348637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.348607 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-5mdj7"] Sep 08 18:38:32.351546 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.351530 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.353850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.353830 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 08 18:38:32.353963 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.353898 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v644g\"" Sep 08 18:38:32.354747 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.354730 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 08 18:38:32.354983 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.354955 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 08 18:38:32.355096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.355022 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 08 18:38:32.360818 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.360791 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-5mdj7"] Sep 08 18:38:32.447597 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.447576 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smpk\" (UniqueName: \"kubernetes.io/projected/75114819-6639-45d0-a7cd-f0bf5defbc8d-kube-api-access-8smpk\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.447705 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.447610 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75114819-6639-45d0-a7cd-f0bf5defbc8d-signing-cabundle\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.447705 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.447641 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75114819-6639-45d0-a7cd-f0bf5defbc8d-signing-key\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.548431 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.548408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smpk\" (UniqueName: \"kubernetes.io/projected/75114819-6639-45d0-a7cd-f0bf5defbc8d-kube-api-access-8smpk\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.548545 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.548442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75114819-6639-45d0-a7cd-f0bf5defbc8d-signing-cabundle\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.548545 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.548473 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75114819-6639-45d0-a7cd-f0bf5defbc8d-signing-key\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.549055 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.549033 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75114819-6639-45d0-a7cd-f0bf5defbc8d-signing-cabundle\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.550707 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.550690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75114819-6639-45d0-a7cd-f0bf5defbc8d-signing-key\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.555775 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.555757 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smpk\" (UniqueName: \"kubernetes.io/projected/75114819-6639-45d0-a7cd-f0bf5defbc8d-kube-api-access-8smpk\") pod \"service-ca-75d99c484-5mdj7\" (UID: \"75114819-6639-45d0-a7cd-f0bf5defbc8d\") " pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.648875 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.648828 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:32.648875 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.648856 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:32.648973 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:32.648935 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:32.649009 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:32.648988 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:34.648975439 +0000 UTC m=+273.158127865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : configmap "kube-rbac-proxy" not found Sep 08 18:38:32.649048 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:32.648938 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:32.649112 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:32.649068 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:34.649055995 +0000 UTC m=+273.158208417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:32.660516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.660500 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-5mdj7" Sep 08 18:38:32.705558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.705531 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ntg6k_2e030b87-5747-4a52-ae8a-1f33fe710f9b/dns-node-resolver/0.log" Sep 08 18:38:32.927292 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:32.927230 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-5mdj7"] Sep 08 18:38:32.930244 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:38:32.930221 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75114819_6639_45d0_a7cd_f0bf5defbc8d.slice/crio-972a135fbd991c3071ea4c76c85a67bb78ba37bd584ed608c1e8b5d121ae5bf5 WatchSource:0}: Error finding container 972a135fbd991c3071ea4c76c85a67bb78ba37bd584ed608c1e8b5d121ae5bf5: Status 404 returned error can't find the container with id 972a135fbd991c3071ea4c76c85a67bb78ba37bd584ed608c1e8b5d121ae5bf5 Sep 08 18:38:33.705968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:33.705940 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5m7lw_728eba11-c5b4-4f60-afed-b980d81a15e6/node-ca/0.log" Sep 08 18:38:33.806348 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:33.806314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-5mdj7" event={"ID":"75114819-6639-45d0-a7cd-f0bf5defbc8d","Type":"ContainerStarted","Data":"972a135fbd991c3071ea4c76c85a67bb78ba37bd584ed608c1e8b5d121ae5bf5"} Sep 08 18:38:34.505977 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:34.505938 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2klms_97d730a5-89d1-4b08-8e23-df1ab8568b99/serve-healthcheck-canary/0.log" Sep 08 18:38:34.664819 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:34.664795 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:34.664920 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:34.664826 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:34.664920 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:34.664916 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:34.665001 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:34.664919 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:34.665001 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:34.664962 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:38.664946405 +0000 UTC m=+277.174098823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:34.665001 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:34.664976 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:38.664969227 +0000 UTC m=+277.174121646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : configmap "kube-rbac-proxy" not found Sep 08 18:38:34.811391 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:34.811326 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-5mdj7" event={"ID":"75114819-6639-45d0-a7cd-f0bf5defbc8d","Type":"ContainerStarted","Data":"6a004b1460421c16ed33852bb5588c5c3133756f8f3e019b0550f0f32a04e2ba"} Sep 08 18:38:34.827250 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:34.827209 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-5mdj7" podStartSLOduration=1.146026677 podStartE2EDuration="2.827194659s" totalImagesPullingTime="1.681167982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:34.826827539 +0000 UTC m=+273.335979982" watchObservedRunningTime="2026-09-08 18:38:34.827194659 +0000 UTC m=+273.336347100" Sep 08 18:38:38.693897 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:38.693857 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:38.693897 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:38.693900 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:38.694347 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:38.693975 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:38.694347 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:38.694041 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:46.694025418 +0000 UTC m=+285.203177842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : configmap "kube-rbac-proxy" not found Sep 08 18:38:38.694347 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:38.694065 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:38.694347 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:38.694117 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls podName:8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:46.694101686 +0000 UTC m=+285.203254120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4dq" (UID: "8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:42.501847 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:42.501812 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-txkkw" podUID="df8f149c-de77-4443-ab61-d9d549496c9a" Sep 08 18:38:42.835041 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:42.834969 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-txkkw" Sep 08 18:38:45.946243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:45.946209 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:38:45.948369 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:45.948345 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df8f149c-de77-4443-ab61-d9d549496c9a-metrics-tls\") pod \"dns-default-txkkw\" (UID: \"df8f149c-de77-4443-ab61-d9d549496c9a\") " pod="openshift-dns/dns-default-txkkw" Sep 08 18:38:46.138034 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.138010 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rsh8d\"" Sep 08 18:38:46.146546 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.146526 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-txkkw" Sep 08 18:38:46.415790 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.415754 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-txkkw"] Sep 08 18:38:46.419203 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:38:46.419178 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8f149c_de77_4443_ab61_d9d549496c9a.slice/crio-22a6e1db9bf35bf35489af60920e83ad08aa8681e0c2f243aa67197e33cb8bed WatchSource:0}: Error finding container 22a6e1db9bf35bf35489af60920e83ad08aa8681e0c2f243aa67197e33cb8bed: Status 404 returned error can't find the container with id 22a6e1db9bf35bf35489af60920e83ad08aa8681e0c2f243aa67197e33cb8bed Sep 08 18:38:46.750934 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.750912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:46.751044 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.750943 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:46.751389 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.751362 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:46.753033 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.753012 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4dq\" (UID: \"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2\") " pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:46.847348 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.847315 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-txkkw" event={"ID":"df8f149c-de77-4443-ab61-d9d549496c9a","Type":"ContainerStarted","Data":"22a6e1db9bf35bf35489af60920e83ad08aa8681e0c2f243aa67197e33cb8bed"} Sep 08 18:38:46.858592 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:46.858576 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9c4dq" Sep 08 18:38:47.136830 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:47.136751 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9c4dq"] Sep 08 18:38:47.140220 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:38:47.140194 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1889d6_325f_4b8f_b5e4_cf7d628cc4f2.slice/crio-bcb2d7d9070dff358d81f00a05565f4115f1a381131d1edb1e66072830d78208 WatchSource:0}: Error finding container bcb2d7d9070dff358d81f00a05565f4115f1a381131d1edb1e66072830d78208: Status 404 returned error can't find the container with id bcb2d7d9070dff358d81f00a05565f4115f1a381131d1edb1e66072830d78208 Sep 08 18:38:47.852719 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:47.852680 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4dq" event={"ID":"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2","Type":"ContainerStarted","Data":"0c8afe108807d1c2626529e8f824f80f59a958aac5829d6ff9a7ad904f0fbb4e"} Sep 08 18:38:47.852884 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:47.852749 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4dq" event={"ID":"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2","Type":"ContainerStarted","Data":"bcb2d7d9070dff358d81f00a05565f4115f1a381131d1edb1e66072830d78208"} Sep 08 18:38:48.858971 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:48.858931 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-txkkw" event={"ID":"df8f149c-de77-4443-ab61-d9d549496c9a","Type":"ContainerStarted","Data":"4f9f76dc73ebbc5ad6c35c7702ad06f19c67e3269cb8125630ecc3087e3e54a8"} Sep 08 18:38:48.859398 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:48.858986 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-txkkw" event={"ID":"df8f149c-de77-4443-ab61-d9d549496c9a","Type":"ContainerStarted","Data":"3e2ae7f0bd0e702e5d503d9977d4ab044105b904d7d424f247bc4716532865d3"} Sep 08 18:38:48.859398 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:48.859019 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-txkkw" Sep 08 18:38:48.860974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:48.860934 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4dq" event={"ID":"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2","Type":"ContainerStarted","Data":"046ad9b42f120ad00a0508e99e914c087278dbb92b4c39df58f84fcbc8ef6402"} Sep 08 18:38:48.878011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:48.877966 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-txkkw" podStartSLOduration=252.365349512 podStartE2EDuration="4m13.877953198s" totalImagesPullingTime="1.512603686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:48.876866112 +0000 UTC m=+287.386018554" watchObservedRunningTime="2026-09-08 18:38:48.877953198 +0000 UTC m=+287.387105638" Sep 08 18:38:49.865851 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:49.865818 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4dq" event={"ID":"8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2","Type":"ContainerStarted","Data":"c93597787c0a28e5775da436b031da65439e3e4054dad0e2de127ea37dc02a68"} Sep 08 18:38:49.880537 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:49.880474 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9c4dq" podStartSLOduration=17.499131519 podStartE2EDuration="19.880460731s" totalImagesPullingTime="2.381329212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:38:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:49.879467362 +0000 UTC m=+288.388619827" watchObservedRunningTime="2026-09-08 18:38:49.880460731 +0000 UTC m=+288.389613171" Sep 08 18:38:51.999371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:51.999337 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-cv8xf"] Sep 08 18:38:52.002283 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.002264 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:38:52.004427 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.004404 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 18:38:52.004427 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.004411 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-r78lv\"" Sep 08 18:38:52.004613 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.004484 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 18:38:52.013189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.013165 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-cv8xf"] Sep 08 18:38:52.083751 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.083726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7c2\" (UniqueName: \"kubernetes.io/projected/e078480b-9e1e-450f-a9e6-6db0ef5c88b3-kube-api-access-dq7c2\") pod \"downloads-85d98c998f-cv8xf\" (UID: \"e078480b-9e1e-450f-a9e6-6db0ef5c88b3\") " pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:38:52.184276 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.184251 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7c2\" (UniqueName: \"kubernetes.io/projected/e078480b-9e1e-450f-a9e6-6db0ef5c88b3-kube-api-access-dq7c2\") pod \"downloads-85d98c998f-cv8xf\" (UID: \"e078480b-9e1e-450f-a9e6-6db0ef5c88b3\") " pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:38:52.193893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.193870 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7c2\" (UniqueName: \"kubernetes.io/projected/e078480b-9e1e-450f-a9e6-6db0ef5c88b3-kube-api-access-dq7c2\") pod \"downloads-85d98c998f-cv8xf\" (UID: \"e078480b-9e1e-450f-a9e6-6db0ef5c88b3\") " pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:38:52.311944 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.311895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:38:52.579110 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.579044 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-cv8xf"] Sep 08 18:38:52.583767 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:38:52.583741 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode078480b_9e1e_450f_a9e6_6db0ef5c88b3.slice/crio-5c84d488552d3a2af5b37fc964fdb37d3eea030da3284cf0fca9932429b2d797 WatchSource:0}: Error finding container 5c84d488552d3a2af5b37fc964fdb37d3eea030da3284cf0fca9932429b2d797: Status 404 returned error can't find the container with id 5c84d488552d3a2af5b37fc964fdb37d3eea030da3284cf0fca9932429b2d797 Sep 08 18:38:52.876485 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:52.876416 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-cv8xf" event={"ID":"e078480b-9e1e-450f-a9e6-6db0ef5c88b3","Type":"ContainerStarted","Data":"5c84d488552d3a2af5b37fc964fdb37d3eea030da3284cf0fca9932429b2d797"} Sep 08 18:38:54.020562 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:38:54.020522 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7rdgz" podUID="04151103-1947-499e-a096-b70859a454ff" Sep 08 18:38:58.868239 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:38:58.868205 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-txkkw" Sep 08 18:39:08.186372 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:08.186348 2557 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 18:39:08.932104 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:08.932060 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-cv8xf" event={"ID":"e078480b-9e1e-450f-a9e6-6db0ef5c88b3","Type":"ContainerStarted","Data":"0d502bdf4f8b7d14c83cefd8ac3bae6793692027170b26a552ed54d138b35416"} Sep 08 18:39:08.932334 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:08.932310 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:39:08.948234 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:08.948185 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-cv8xf" podStartSLOduration=2.282073062 podStartE2EDuration="17.948171955s" totalImagesPullingTime="15.666098893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:08.946704188 +0000 UTC m=+307.455856630" watchObservedRunningTime="2026-09-08 18:39:08.948171955 +0000 UTC m=+307.457324396" Sep 08 18:39:08.950905 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:08.950879 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-cv8xf" Sep 08 18:39:09.018931 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.018895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:39:09.460043 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.460003 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-drp29"] Sep 08 18:39:09.464729 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.464704 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.467824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.467798 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 08 18:39:09.467947 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.467841 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-j5b9j\"" Sep 08 18:39:09.468130 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.468112 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 18:39:09.468232 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.468209 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 18:39:09.468329 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.468306 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 08 18:39:09.468448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.468113 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 18:39:09.472195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.472171 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-drp29"] Sep 08 18:39:09.493459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.493387 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d7qsn"] Sep 08 18:39:09.496951 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.496932 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.499253 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.499218 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rfb7z\"" Sep 08 18:39:09.499554 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.499538 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 18:39:09.499822 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.499808 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 18:39:09.500002 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.499986 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 18:39:09.613794 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.613761 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-textfile\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614066 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614034 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614151 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614080 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-metrics-client-ca\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614151 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614105 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614157 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk8kc\" (UniqueName: \"kubernetes.io/projected/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-kube-api-access-qk8kc\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.614257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614215 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-tls\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.614448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614271 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca7316c5-1fec-450c-a9da-4429f14e0075-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.614448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614296 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwd9t\" (UniqueName: \"kubernetes.io/projected/ca7316c5-1fec-450c-a9da-4429f14e0075-kube-api-access-kwd9t\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.614448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614322 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-wtmp\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-sys\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.614448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.614401 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-root\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.714999 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.714923 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-metrics-client-ca\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.715799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.715778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.715976 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.715962 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk8kc\" (UniqueName: \"kubernetes.io/projected/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-kube-api-access-qk8kc\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.716075 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716061 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.716175 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-tls\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.716265 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716251 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.716352 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca7316c5-1fec-450c-a9da-4429f14e0075-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.716444 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd9t\" (UniqueName: \"kubernetes.io/projected/ca7316c5-1fec-450c-a9da-4429f14e0075-kube-api-access-kwd9t\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.716530 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-wtmp\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.716660 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.716632 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-sys\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.717871 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.717838 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-root\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.715730 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-metrics-client-ca\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.717239 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-wtmp\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.717275 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-sys\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.717435 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca7316c5-1fec-450c-a9da-4429f14e0075-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:09.716669 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:09.718233 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-tls podName:eed21ef4-27c7-4dfb-a601-952fe1c7b1cc nodeName:}" failed. No retries permitted until 2026-09-08 18:39:10.218214171 +0000 UTC m=+308.727366599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-tls") pod "node-exporter-d7qsn" (UID: "eed21ef4-27c7-4dfb-a601-952fe1c7b1cc") : secret "node-exporter-tls" not found Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.718027 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-root\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:09.716985 2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 08 18:39:09.718503 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:09.718473 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls podName:ca7316c5-1fec-450c-a9da-4429f14e0075 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:10.2184594 +0000 UTC m=+308.727611823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-drp29" (UID: "ca7316c5-1fec-450c-a9da-4429f14e0075") : secret "openshift-state-metrics-tls" not found Sep 08 18:39:09.719283 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.719246 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-textfile\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.719384 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.719368 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-textfile\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.719501 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.719488 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.720072 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.720000 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.728659 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.723543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.729548 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.727792 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk8kc\" (UniqueName: \"kubernetes.io/projected/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-kube-api-access-qk8kc\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:09.732261 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.732213 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwd9t\" (UniqueName: \"kubernetes.io/projected/ca7316c5-1fec-450c-a9da-4429f14e0075-kube-api-access-kwd9t\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:09.735658 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:09.734984 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:10.223485 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:10.223447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-tls\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:10.223695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:10.223490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:10.223695 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:10.223609 2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 08 18:39:10.223841 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:10.223714 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls podName:ca7316c5-1fec-450c-a9da-4429f14e0075 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:11.223693295 +0000 UTC m=+309.732845720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-drp29" (UID: "ca7316c5-1fec-450c-a9da-4429f14e0075") : secret "openshift-state-metrics-tls" not found Sep 08 18:39:10.226306 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:10.226278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eed21ef4-27c7-4dfb-a601-952fe1c7b1cc-node-exporter-tls\") pod \"node-exporter-d7qsn\" (UID: \"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc\") " pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:10.408809 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:10.408771 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7qsn" Sep 08 18:39:10.454392 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:10.453057 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:39:10.943096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:10.943046 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7qsn" event={"ID":"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc","Type":"ContainerStarted","Data":"c05cacc5ce475b11b7f6074dbf3c399d2468a9f6ee7f1bbafddbbd59f019706f"} Sep 08 18:39:11.232896 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.232717 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:11.235673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.235629 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca7316c5-1fec-450c-a9da-4429f14e0075-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-drp29\" (UID: \"ca7316c5-1fec-450c-a9da-4429f14e0075\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:11.277564 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.277534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" Sep 08 18:39:11.458691 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.458654 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69bdcc9b79-76hmc"] Sep 08 18:39:11.486067 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.485997 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69bdcc9b79-76hmc"] Sep 08 18:39:11.486215 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.486117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.488486 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.488460 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 08 18:39:11.488796 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.488773 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 08 18:39:11.488910 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.488834 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 08 18:39:11.489056 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.488944 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1l9o9s8edmse9\"" Sep 08 18:39:11.489140 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.489080 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 08 18:39:11.489203 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.489160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 08 18:39:11.489203 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.489178 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r2mtp\"" Sep 08 18:39:11.635176 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635140 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635313 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635181 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-grpc-tls\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635313 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635202 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-tls\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635313 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635218 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635313 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635283 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9pp\" (UniqueName: \"kubernetes.io/projected/731d1f4d-303e-44f0-9de0-db0f45c30e97-kube-api-access-2g9pp\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635570 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635358 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635570 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635428 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.635570 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.635472 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731d1f4d-303e-44f0-9de0-db0f45c30e97-metrics-client-ca\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736759 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736759 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731d1f4d-303e-44f0-9de0-db0f45c30e97-metrics-client-ca\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-grpc-tls\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-tls\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736880 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9pp\" (UniqueName: \"kubernetes.io/projected/731d1f4d-303e-44f0-9de0-db0f45c30e97-kube-api-access-2g9pp\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.736957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.736921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.737525 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.737472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731d1f4d-303e-44f0-9de0-db0f45c30e97-metrics-client-ca\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.740703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.740090 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.740703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.740618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.740703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.740637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-grpc-tls\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.740957 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.740912 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.742166 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.742142 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.742278 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.742196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/731d1f4d-303e-44f0-9de0-db0f45c30e97-secret-thanos-querier-tls\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.744882 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.744861 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9pp\" (UniqueName: \"kubernetes.io/projected/731d1f4d-303e-44f0-9de0-db0f45c30e97-kube-api-access-2g9pp\") pod \"thanos-querier-69bdcc9b79-76hmc\" (UID: \"731d1f4d-303e-44f0-9de0-db0f45c30e97\") " pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.797337 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.797309 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:11.823506 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:11.823481 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-drp29"] Sep 08 18:39:11.943801 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:39:11.943773 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7316c5_1fec_450c_a9da_4429f14e0075.slice/crio-de6f5a1959a83ec26d978f609681bb0814986ab225f91b24ea9b6f0f2074cefc WatchSource:0}: Error finding container de6f5a1959a83ec26d978f609681bb0814986ab225f91b24ea9b6f0f2074cefc: Status 404 returned error can't find the container with id de6f5a1959a83ec26d978f609681bb0814986ab225f91b24ea9b6f0f2074cefc Sep 08 18:39:12.252099 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.252041 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69bdcc9b79-76hmc"] Sep 08 18:39:12.266053 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:39:12.265974 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731d1f4d_303e_44f0_9de0_db0f45c30e97.slice/crio-54614238ef58ed8725c2d76b1243b02d64d00737e4ef9fa9b119ecf061cad6ab WatchSource:0}: Error finding container 54614238ef58ed8725c2d76b1243b02d64d00737e4ef9fa9b119ecf061cad6ab: Status 404 returned error can't find the container with id 54614238ef58ed8725c2d76b1243b02d64d00737e4ef9fa9b119ecf061cad6ab Sep 08 18:39:12.955400 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.955350 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" event={"ID":"ca7316c5-1fec-450c-a9da-4429f14e0075","Type":"ContainerStarted","Data":"ad7d441981d6a89a2d1a30fb29de21c2a1297a6a87c4f9eb8054fe53d5c5a4d3"} Sep 08 18:39:12.955926 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.955423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" event={"ID":"ca7316c5-1fec-450c-a9da-4429f14e0075","Type":"ContainerStarted","Data":"088112a7dd60859f5fc644943217273cbdeef7c0856d934ad171682590a9d325"} Sep 08 18:39:12.955926 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.955450 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" event={"ID":"ca7316c5-1fec-450c-a9da-4429f14e0075","Type":"ContainerStarted","Data":"de6f5a1959a83ec26d978f609681bb0814986ab225f91b24ea9b6f0f2074cefc"} Sep 08 18:39:12.958598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.958514 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"54614238ef58ed8725c2d76b1243b02d64d00737e4ef9fa9b119ecf061cad6ab"} Sep 08 18:39:12.963846 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.963809 2557 generic.go:356] "Generic (PLEG): container finished" podID="eed21ef4-27c7-4dfb-a601-952fe1c7b1cc" containerID="fb24d3571981a9f57f0f691a16561044cf4448c2bc711423ff167a6e963b5ff0" exitCode=0 Sep 08 18:39:12.964171 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:12.963890 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7qsn" event={"ID":"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc","Type":"ContainerDied","Data":"fb24d3571981a9f57f0f691a16561044cf4448c2bc711423ff167a6e963b5ff0"} Sep 08 18:39:13.969934 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:13.969892 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7qsn" event={"ID":"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc","Type":"ContainerStarted","Data":"8cd720a622d80a6b18d538e0dd5fdae56bb25c19a84dd294025e84d6f4c8c402"} Sep 08 18:39:13.970364 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:13.969953 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7qsn" event={"ID":"eed21ef4-27c7-4dfb-a601-952fe1c7b1cc","Type":"ContainerStarted","Data":"a2cb97be61f56c215d7db229032e90559615650619cbc4aa6ff8a6abfe1d53fb"} Sep 08 18:39:13.987620 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:13.987574 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d7qsn" podStartSLOduration=3.381240435 podStartE2EDuration="4.987551761s" totalImagesPullingTime="1.517768581s" totalInitContainerRuntime="88.542745ms" isStatefulPod=true podCreationTimestamp="2026-09-08 18:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:13.986694353 +0000 UTC m=+312.495846793" watchObservedRunningTime="2026-09-08 18:39:13.987551761 +0000 UTC m=+312.496704203" Sep 08 18:39:14.258178 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.258149 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf"] Sep 08 18:39:14.279180 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.279146 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf"] Sep 08 18:39:14.279327 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.279191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.281532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.281360 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 08 18:39:14.281532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.281367 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8k2pj\"" Sep 08 18:39:14.361676 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.361607 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqgd\" (UniqueName: \"kubernetes.io/projected/4b92bd2d-9eb8-4b95-9188-86ecfec03604-kube-api-access-xbqgd\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.361830 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.361694 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b92bd2d-9eb8-4b95-9188-86ecfec03604-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.462600 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.462574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqgd\" (UniqueName: \"kubernetes.io/projected/4b92bd2d-9eb8-4b95-9188-86ecfec03604-kube-api-access-xbqgd\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.462724 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.462636 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b92bd2d-9eb8-4b95-9188-86ecfec03604-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.462801 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:14.462781 2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 08 18:39:14.462911 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:39:14.462840 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b92bd2d-9eb8-4b95-9188-86ecfec03604-monitoring-plugin-cert podName:4b92bd2d-9eb8-4b95-9188-86ecfec03604 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:14.962821479 +0000 UTC m=+313.471973903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4b92bd2d-9eb8-4b95-9188-86ecfec03604-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-cnblf" (UID: "4b92bd2d-9eb8-4b95-9188-86ecfec03604") : secret "monitoring-plugin-cert" not found Sep 08 18:39:14.470524 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.470496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqgd\" (UniqueName: \"kubernetes.io/projected/4b92bd2d-9eb8-4b95-9188-86ecfec03604-kube-api-access-xbqgd\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.966345 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.966306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b92bd2d-9eb8-4b95-9188-86ecfec03604-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.969361 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.969335 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b92bd2d-9eb8-4b95-9188-86ecfec03604-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cnblf\" (UID: \"4b92bd2d-9eb8-4b95-9188-86ecfec03604\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:14.979914 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.979379 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" event={"ID":"ca7316c5-1fec-450c-a9da-4429f14e0075","Type":"ContainerStarted","Data":"dec471cbf19803179b5ca4e31a5afc89b15a9f9c9c2cb4ac9c93f2830118a1b8"} Sep 08 18:39:14.995660 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:14.995590 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-drp29" podStartSLOduration=3.895285827 podStartE2EDuration="5.995571583s" totalImagesPullingTime="2.100285756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:14.993752659 +0000 UTC m=+313.502905110" watchObservedRunningTime="2026-09-08 18:39:14.995571583 +0000 UTC m=+313.504724026" Sep 08 18:39:15.190970 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:15.190942 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:16.031288 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.031205 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf"] Sep 08 18:39:16.174638 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:39:16.174603 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b92bd2d_9eb8_4b95_9188_86ecfec03604.slice/crio-39c56dd0a5c9c7f15a9a233898c41964ec1e23ad1db60d36db80bb81c0bc7153 WatchSource:0}: Error finding container 39c56dd0a5c9c7f15a9a233898c41964ec1e23ad1db60d36db80bb81c0bc7153: Status 404 returned error can't find the container with id 39c56dd0a5c9c7f15a9a233898c41964ec1e23ad1db60d36db80bb81c0bc7153 Sep 08 18:39:16.437375 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.437350 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fddb79c5d-sgpjf"] Sep 08 18:39:16.458125 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.458101 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fddb79c5d-sgpjf"] Sep 08 18:39:16.458243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.458201 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.460464 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.460440 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 18:39:16.460610 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.460590 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 18:39:16.460709 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.460609 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 18:39:16.461114 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.461095 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 18:39:16.461114 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.461107 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vd5bl\"" Sep 08 18:39:16.461262 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.461156 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 18:39:16.465544 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.465524 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 18:39:16.579440 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-service-ca\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.579543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579444 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-trusted-ca-bundle\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.579543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579478 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-serving-cert\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.579543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579499 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-oauth-serving-cert\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.579717 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579581 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-oauth-config\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.579717 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st475\" (UniqueName: \"kubernetes.io/projected/a1821cdb-59d3-4899-9d96-611a4ab50295-kube-api-access-st475\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.579717 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.579698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-console-config\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681000 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-service-ca\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681155 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681046 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-trusted-ca-bundle\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681155 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681106 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-serving-cert\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681296 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-oauth-serving-cert\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-oauth-config\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681363 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st475\" (UniqueName: \"kubernetes.io/projected/a1821cdb-59d3-4899-9d96-611a4ab50295-kube-api-access-st475\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-console-config\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.681982 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.681898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-service-ca\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.682072 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.682007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-oauth-serving-cert\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.682072 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.682004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-trusted-ca-bundle\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.682201 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.682177 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-console-config\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.684138 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.684115 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-serving-cert\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.684236 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.684221 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-oauth-config\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.689050 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.689028 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st475\" (UniqueName: \"kubernetes.io/projected/a1821cdb-59d3-4899-9d96-611a4ab50295-kube-api-access-st475\") pod \"console-7fddb79c5d-sgpjf\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.769009 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.768980 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:16.989744 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.989665 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" event={"ID":"4b92bd2d-9eb8-4b95-9188-86ecfec03604","Type":"ContainerStarted","Data":"39c56dd0a5c9c7f15a9a233898c41964ec1e23ad1db60d36db80bb81c0bc7153"} Sep 08 18:39:16.992805 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.992762 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"88adbd32c6563054ea610b04e79aeba10125713288efc44f94af3e887f272fee"} Sep 08 18:39:16.992946 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.992839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"dbe342ad6e91352cb786621a4f469da6f7547ef084ae73da9bc5c8552377256a"} Sep 08 18:39:16.992946 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:16.992867 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"f4d5afb317caed4d6682f4f3ed4795b028aa6824b6a24edbdb918231650771e3"} Sep 08 18:39:17.074799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:17.074693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fddb79c5d-sgpjf"] Sep 08 18:39:17.078572 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:39:17.078532 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1821cdb_59d3_4899_9d96_611a4ab50295.slice/crio-9a40aba0617cda53a2394f937715474fc890b1811724f8fe0cf788ec635d7003 WatchSource:0}: Error finding container 9a40aba0617cda53a2394f937715474fc890b1811724f8fe0cf788ec635d7003: Status 404 returned error can't find the container with id 9a40aba0617cda53a2394f937715474fc890b1811724f8fe0cf788ec635d7003 Sep 08 18:39:17.997893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:17.997848 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fddb79c5d-sgpjf" event={"ID":"a1821cdb-59d3-4899-9d96-611a4ab50295","Type":"ContainerStarted","Data":"9a40aba0617cda53a2394f937715474fc890b1811724f8fe0cf788ec635d7003"} Sep 08 18:39:19.003196 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.003166 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" event={"ID":"4b92bd2d-9eb8-4b95-9188-86ecfec03604","Type":"ContainerStarted","Data":"6f55aa657b4a3da98b415cebe39dcf9b50a9a9acf04439eee4382252aea82313"} Sep 08 18:39:19.003629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.003257 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:19.006710 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.006683 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"eb2f9b3ebbd8a2bcb2f725101e046dc105c69b3b6f758a0646b233cb65711cd4"} Sep 08 18:39:19.006819 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.006734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"539254746e05dcb113c7c765ca6723b392c4a1523c809fe552ea8befb6d91507"} Sep 08 18:39:19.006819 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.006767 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" event={"ID":"731d1f4d-303e-44f0-9de0-db0f45c30e97","Type":"ContainerStarted","Data":"529d77167833b64b4c0c3f0967377f06ab130dbf41729c949a0a275967cf57fa"} Sep 08 18:39:19.006819 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.006796 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:19.009089 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.009065 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" Sep 08 18:39:19.016336 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.016289 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cnblf" podStartSLOduration=2.913599026 podStartE2EDuration="5.016277379s" totalImagesPullingTime="2.102678353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:19.016061263 +0000 UTC m=+317.525213708" watchObservedRunningTime="2026-09-08 18:39:19.016277379 +0000 UTC m=+317.525429819" Sep 08 18:39:19.048832 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:19.048791 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" podStartSLOduration=2.408685619 podStartE2EDuration="8.048778123s" totalImagesPullingTime="5.640092504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:19.04856999 +0000 UTC m=+317.557722432" watchObservedRunningTime="2026-09-08 18:39:19.048778123 +0000 UTC m=+317.557930566" Sep 08 18:39:22.023806 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:22.023774 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fddb79c5d-sgpjf" event={"ID":"a1821cdb-59d3-4899-9d96-611a4ab50295","Type":"ContainerStarted","Data":"60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4"} Sep 08 18:39:22.036281 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:22.036237 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fddb79c5d-sgpjf" podStartSLOduration=1.9873622050000002 podStartE2EDuration="6.036221831s" totalImagesPullingTime="4.048859626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:22.035628896 +0000 UTC m=+320.544781350" watchObservedRunningTime="2026-09-08 18:39:22.036221831 +0000 UTC m=+320.545374270" Sep 08 18:39:25.016849 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:25.016824 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69bdcc9b79-76hmc" Sep 08 18:39:26.769695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:26.769663 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:26.770163 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:26.769722 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:26.774126 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:26.774102 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:27.038666 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:27.038596 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:39:36.306705 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:36.306638 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" podUID="d8e0ebd8-3620-4114-b4e6-b43077c9c6c3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 18:39:41.081871 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:41.081829 2557 generic.go:356] "Generic (PLEG): container finished" podID="b459ecd6-967b-4efe-9567-c782a158e362" containerID="fcd5755b9d3efd3fbedc5ea7626fb068d92c744c801c333b4303f0d6276d8593" exitCode=0 Sep 08 18:39:41.082244 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:41.081879 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" event={"ID":"b459ecd6-967b-4efe-9567-c782a158e362","Type":"ContainerDied","Data":"fcd5755b9d3efd3fbedc5ea7626fb068d92c744c801c333b4303f0d6276d8593"} Sep 08 18:39:41.082244 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:41.082153 2557 scope.go:117] "RemoveContainer" containerID="fcd5755b9d3efd3fbedc5ea7626fb068d92c744c801c333b4303f0d6276d8593" Sep 08 18:39:42.086767 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:42.086740 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hdx4b" event={"ID":"b459ecd6-967b-4efe-9567-c782a158e362","Type":"ContainerStarted","Data":"3a2d97c7805c555872e871ae8efe562fd71453c6d8c1eb12b3c50d0279f95a7e"} Sep 08 18:39:46.306482 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:46.306445 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" podUID="d8e0ebd8-3620-4114-b4e6-b43077c9c6c3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 18:39:56.306190 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:56.306104 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" podUID="d8e0ebd8-3620-4114-b4e6-b43077c9c6c3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 18:39:56.306190 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:56.306183 2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" Sep 08 18:39:56.306606 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:56.306577 2557 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b9a6397d9bd75605ca9f2beba19ec49cde361b2f4a51b0f514c78ccde142e9ca"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 08 18:39:56.306671 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:56.306626 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" podUID="d8e0ebd8-3620-4114-b4e6-b43077c9c6c3" containerName="service-proxy" containerID="cri-o://b9a6397d9bd75605ca9f2beba19ec49cde361b2f4a51b0f514c78ccde142e9ca" gracePeriod=30 Sep 08 18:39:57.135416 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:57.135385 2557 generic.go:356] "Generic (PLEG): container finished" podID="d8e0ebd8-3620-4114-b4e6-b43077c9c6c3" containerID="b9a6397d9bd75605ca9f2beba19ec49cde361b2f4a51b0f514c78ccde142e9ca" exitCode=2 Sep 08 18:39:57.135564 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:57.135442 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" event={"ID":"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3","Type":"ContainerDied","Data":"b9a6397d9bd75605ca9f2beba19ec49cde361b2f4a51b0f514c78ccde142e9ca"} Sep 08 18:39:57.135564 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:39:57.135479 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f968865-vlcbv" event={"ID":"d8e0ebd8-3620-4114-b4e6-b43077c9c6c3","Type":"ContainerStarted","Data":"054a5acb383db3071fe00c0eb9ea278fb838b8806115e1b442917fc20313dcc2"} Sep 08 18:40:15.983378 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:15.983342 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:40:15.985459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:15.985442 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04151103-1947-499e-a096-b70859a454ff-metrics-certs\") pod \"network-metrics-daemon-7rdgz\" (UID: \"04151103-1947-499e-a096-b70859a454ff\") " pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:40:16.221582 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:16.221554 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z79bg\"" Sep 08 18:40:16.230280 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:16.230259 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rdgz" Sep 08 18:40:16.499007 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:16.498936 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7rdgz"] Sep 08 18:40:16.502859 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:40:16.502835 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04151103_1947_499e_a096_b70859a454ff.slice/crio-0dc694bb2636b5bc197acc8908867b087ec55d7b99e917680a68de5da68821ff WatchSource:0}: Error finding container 0dc694bb2636b5bc197acc8908867b087ec55d7b99e917680a68de5da68821ff: Status 404 returned error can't find the container with id 0dc694bb2636b5bc197acc8908867b087ec55d7b99e917680a68de5da68821ff Sep 08 18:40:17.204488 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:17.204445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rdgz" event={"ID":"04151103-1947-499e-a096-b70859a454ff","Type":"ContainerStarted","Data":"0dc694bb2636b5bc197acc8908867b087ec55d7b99e917680a68de5da68821ff"} Sep 08 18:40:18.210259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:18.210224 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rdgz" event={"ID":"04151103-1947-499e-a096-b70859a454ff","Type":"ContainerStarted","Data":"07b4e4a43edb848a3c72149b7bff060e5f26b0443c1c212abcd55380226c35bd"} Sep 08 18:40:18.210602 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:18.210284 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rdgz" event={"ID":"04151103-1947-499e-a096-b70859a454ff","Type":"ContainerStarted","Data":"5a42171f56958695a7a534d9d40b1bb31e2d5365e42661c66860559fa69f7c6c"} Sep 08 18:40:18.224373 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:18.224324 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7rdgz" podStartSLOduration=375.160877912 podStartE2EDuration="6m16.224311103s" totalImagesPullingTime="1.063433191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:18.223672567 +0000 UTC m=+376.732825009" watchObservedRunningTime="2026-09-08 18:40:18.224311103 +0000 UTC m=+376.733463575" Sep 08 18:40:48.064433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:40:48.064401 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fddb79c5d-sgpjf"] Sep 08 18:41:13.083418 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.083361 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fddb79c5d-sgpjf" podUID="a1821cdb-59d3-4899-9d96-611a4ab50295" containerName="console" containerID="cri-o://60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4" gracePeriod=15 Sep 08 18:41:13.319936 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.319914 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:41:13.402429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.402368 2557 generic.go:356] "Generic (PLEG): container finished" podID="a1821cdb-59d3-4899-9d96-611a4ab50295" containerID="60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4" exitCode=0 Sep 08 18:41:13.402429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.402407 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fddb79c5d-sgpjf" Sep 08 18:41:13.402562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.402442 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fddb79c5d-sgpjf" event={"ID":"a1821cdb-59d3-4899-9d96-611a4ab50295","Type":"ContainerDied","Data":"60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4"} Sep 08 18:41:13.402562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.402482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fddb79c5d-sgpjf" event={"ID":"a1821cdb-59d3-4899-9d96-611a4ab50295","Type":"ContainerDied","Data":"9a40aba0617cda53a2394f937715474fc890b1811724f8fe0cf788ec635d7003"} Sep 08 18:41:13.402562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.402506 2557 scope.go:117] "RemoveContainer" containerID="60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4" Sep 08 18:41:13.410790 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.410774 2557 scope.go:117] "RemoveContainer" containerID="60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4" Sep 08 18:41:13.411038 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:41:13.411021 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4\": container with ID starting with 60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4 not found: ID does not exist" containerID="60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4" Sep 08 18:41:13.411087 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.411047 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4"} err="failed to get container status \"60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4\": rpc error: code = NotFound desc = could not find container \"60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4\": container with ID starting with 60d8d61b95c9d79660b4e1688def5d845e8a77c15b3c8d924b97b3f9a75fb0d4 not found: ID does not exist" Sep 08 18:41:13.456292 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456262 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-oauth-serving-cert\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456374 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456300 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-oauth-config\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456374 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456318 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-serving-cert\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456374 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456344 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-console-config\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456528 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456373 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-service-ca\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456528 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456430 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st475\" (UniqueName: \"kubernetes.io/projected/a1821cdb-59d3-4899-9d96-611a4ab50295-kube-api-access-st475\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456528 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456477 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-trusted-ca-bundle\") pod \"a1821cdb-59d3-4899-9d96-611a4ab50295\" (UID: \"a1821cdb-59d3-4899-9d96-611a4ab50295\") " Sep 08 18:41:13.456703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456577 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:41:13.456756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456727 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-oauth-serving-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.456849 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456824 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-console-config" (OuterVolumeSpecName: "console-config") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:41:13.456917 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456882 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:41:13.456977 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.456954 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:41:13.458460 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.458431 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1821cdb-59d3-4899-9d96-611a4ab50295-kube-api-access-st475" (OuterVolumeSpecName: "kube-api-access-st475") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "kube-api-access-st475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:41:13.458547 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.458466 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:41:13.458547 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.458487 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1821cdb-59d3-4899-9d96-611a4ab50295" (UID: "a1821cdb-59d3-4899-9d96-611a4ab50295"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:41:13.557858 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.557823 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st475\" (UniqueName: \"kubernetes.io/projected/a1821cdb-59d3-4899-9d96-611a4ab50295-kube-api-access-st475\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.557858 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.557858 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-trusted-ca-bundle\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.557974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.557868 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-oauth-config\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.557974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.557877 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1821cdb-59d3-4899-9d96-611a4ab50295-console-serving-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.557974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.557885 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-console-config\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.557974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.557893 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1821cdb-59d3-4899-9d96-611a4ab50295-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:41:13.722001 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.721977 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fddb79c5d-sgpjf"] Sep 08 18:41:13.725123 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:13.725101 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fddb79c5d-sgpjf"] Sep 08 18:41:14.023677 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:41:14.023635 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1821cdb-59d3-4899-9d96-611a4ab50295" path="/var/lib/kubelet/pods/a1821cdb-59d3-4899-9d96-611a4ab50295/volumes" Sep 08 18:43:40.360851 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.360820 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf"] Sep 08 18:43:40.361324 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.361074 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1821cdb-59d3-4899-9d96-611a4ab50295" containerName="console" Sep 08 18:43:40.361324 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.361085 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1821cdb-59d3-4899-9d96-611a4ab50295" containerName="console" Sep 08 18:43:40.361324 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.361125 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1821cdb-59d3-4899-9d96-611a4ab50295" containerName="console" Sep 08 18:43:40.363778 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.363762 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.366157 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.366124 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-sg66v\"" Sep 08 18:43:40.366303 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.366159 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:43:40.366303 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.366187 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 08 18:43:40.377189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.377166 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf"] Sep 08 18:43:40.503921 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.503901 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eaa15c3-b41c-414f-9e0b-77e948ed346e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jk6wf\" (UID: \"4eaa15c3-b41c-414f-9e0b-77e948ed346e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.504018 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.503928 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhds\" (UniqueName: \"kubernetes.io/projected/4eaa15c3-b41c-414f-9e0b-77e948ed346e-kube-api-access-xhhds\") pod \"cert-manager-operator-controller-manager-55c444bf86-jk6wf\" (UID: \"4eaa15c3-b41c-414f-9e0b-77e948ed346e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.604516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.604495 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eaa15c3-b41c-414f-9e0b-77e948ed346e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jk6wf\" (UID: \"4eaa15c3-b41c-414f-9e0b-77e948ed346e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.604607 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.604525 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhds\" (UniqueName: \"kubernetes.io/projected/4eaa15c3-b41c-414f-9e0b-77e948ed346e-kube-api-access-xhhds\") pod \"cert-manager-operator-controller-manager-55c444bf86-jk6wf\" (UID: \"4eaa15c3-b41c-414f-9e0b-77e948ed346e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.604862 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.604845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eaa15c3-b41c-414f-9e0b-77e948ed346e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jk6wf\" (UID: \"4eaa15c3-b41c-414f-9e0b-77e948ed346e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.613046 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.612991 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhds\" (UniqueName: \"kubernetes.io/projected/4eaa15c3-b41c-414f-9e0b-77e948ed346e-kube-api-access-xhhds\") pod \"cert-manager-operator-controller-manager-55c444bf86-jk6wf\" (UID: \"4eaa15c3-b41c-414f-9e0b-77e948ed346e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.672937 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.672917 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" Sep 08 18:43:40.946892 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:40.946781 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf"] Sep 08 18:43:40.951700 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:43:40.951673 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaa15c3_b41c_414f_9e0b_77e948ed346e.slice/crio-08d7ce863c27ab185bd64a96e52f5e42314acf36e7d2dfa0ea04e0220dd07501 WatchSource:0}: Error finding container 08d7ce863c27ab185bd64a96e52f5e42314acf36e7d2dfa0ea04e0220dd07501: Status 404 returned error can't find the container with id 08d7ce863c27ab185bd64a96e52f5e42314acf36e7d2dfa0ea04e0220dd07501 Sep 08 18:43:41.906357 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:41.906323 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" event={"ID":"4eaa15c3-b41c-414f-9e0b-77e948ed346e","Type":"ContainerStarted","Data":"08d7ce863c27ab185bd64a96e52f5e42314acf36e7d2dfa0ea04e0220dd07501"} Sep 08 18:43:44.921887 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:44.921845 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" event={"ID":"4eaa15c3-b41c-414f-9e0b-77e948ed346e","Type":"ContainerStarted","Data":"6d45abcb7bce062779fb4f011355a8f8228005433e3cfd6f78cd8ebc8ef62550"} Sep 08 18:43:44.949469 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:44.949419 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jk6wf" podStartSLOduration=1.540209221 podStartE2EDuration="4.949403742s" totalImagesPullingTime="3.409194521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:44.948380572 +0000 UTC m=+583.457533013" watchObservedRunningTime="2026-09-08 18:43:44.949403742 +0000 UTC m=+583.458556183" Sep 08 18:43:57.711893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.711861 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n"] Sep 08 18:43:57.714562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.714546 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:57.717096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.717073 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 18:43:57.717206 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.717125 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7vfvv\"" Sep 08 18:43:57.717787 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.717773 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:43:57.721850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.721830 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n"] Sep 08 18:43:57.817778 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.817755 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74bfae71-10e3-49ed-8618-7dbd81d829a3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pf84n\" (UID: \"74bfae71-10e3-49ed-8618-7dbd81d829a3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:57.817874 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.817806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8xk\" (UniqueName: \"kubernetes.io/projected/74bfae71-10e3-49ed-8618-7dbd81d829a3-kube-api-access-bk8xk\") pod \"openshift-lws-operator-fd8ccff4c-pf84n\" (UID: \"74bfae71-10e3-49ed-8618-7dbd81d829a3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:57.918891 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.918866 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74bfae71-10e3-49ed-8618-7dbd81d829a3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pf84n\" (UID: \"74bfae71-10e3-49ed-8618-7dbd81d829a3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:57.918974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.918909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8xk\" (UniqueName: \"kubernetes.io/projected/74bfae71-10e3-49ed-8618-7dbd81d829a3-kube-api-access-bk8xk\") pod \"openshift-lws-operator-fd8ccff4c-pf84n\" (UID: \"74bfae71-10e3-49ed-8618-7dbd81d829a3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:57.919272 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.919255 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74bfae71-10e3-49ed-8618-7dbd81d829a3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pf84n\" (UID: \"74bfae71-10e3-49ed-8618-7dbd81d829a3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:57.926748 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:57.926729 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8xk\" (UniqueName: \"kubernetes.io/projected/74bfae71-10e3-49ed-8618-7dbd81d829a3-kube-api-access-bk8xk\") pod \"openshift-lws-operator-fd8ccff4c-pf84n\" (UID: \"74bfae71-10e3-49ed-8618-7dbd81d829a3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:58.023577 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:58.023559 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" Sep 08 18:43:58.291036 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:58.290870 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n"] Sep 08 18:43:58.293850 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:43:58.293819 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74bfae71_10e3_49ed_8618_7dbd81d829a3.slice/crio-fa39b72ae7746136e82afbc292565a10f9f3e27fd4355a309485ae38d533e24f WatchSource:0}: Error finding container fa39b72ae7746136e82afbc292565a10f9f3e27fd4355a309485ae38d533e24f: Status 404 returned error can't find the container with id fa39b72ae7746136e82afbc292565a10f9f3e27fd4355a309485ae38d533e24f Sep 08 18:43:58.978506 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:43:58.978444 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" event={"ID":"74bfae71-10e3-49ed-8618-7dbd81d829a3","Type":"ContainerStarted","Data":"fa39b72ae7746136e82afbc292565a10f9f3e27fd4355a309485ae38d533e24f"} Sep 08 18:44:00.989528 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:00.989493 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" event={"ID":"74bfae71-10e3-49ed-8618-7dbd81d829a3","Type":"ContainerStarted","Data":"15f2e446e1ec8aa594e6b30e1daf6d51b3a01d3b8964e40e89c282586ca6b6df"} Sep 08 18:44:01.006997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:01.006951 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pf84n" podStartSLOduration=1.8363461920000002 podStartE2EDuration="4.006936486s" totalImagesPullingTime="2.170590294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:01.00612706 +0000 UTC m=+599.515279503" watchObservedRunningTime="2026-09-08 18:44:01.006936486 +0000 UTC m=+599.516088928" Sep 08 18:44:05.416306 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.416274 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ps8sw"] Sep 08 18:44:05.422248 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.422225 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.424811 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.424791 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gpkwg\"" Sep 08 18:44:05.425416 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.425401 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 18:44:05.425754 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.425740 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 18:44:05.429597 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.429574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ps8sw"] Sep 08 18:44:05.572173 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.572151 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71c4c37e-08cc-439c-9500-040c33d3fe94-bound-sa-token\") pod \"cert-manager-55b4d954d5-ps8sw\" (UID: \"71c4c37e-08cc-439c-9500-040c33d3fe94\") " pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.572270 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.572194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkw9w\" (UniqueName: \"kubernetes.io/projected/71c4c37e-08cc-439c-9500-040c33d3fe94-kube-api-access-lkw9w\") pod \"cert-manager-55b4d954d5-ps8sw\" (UID: \"71c4c37e-08cc-439c-9500-040c33d3fe94\") " pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.672873 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.672821 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71c4c37e-08cc-439c-9500-040c33d3fe94-bound-sa-token\") pod \"cert-manager-55b4d954d5-ps8sw\" (UID: \"71c4c37e-08cc-439c-9500-040c33d3fe94\") " pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.672873 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.672864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkw9w\" (UniqueName: \"kubernetes.io/projected/71c4c37e-08cc-439c-9500-040c33d3fe94-kube-api-access-lkw9w\") pod \"cert-manager-55b4d954d5-ps8sw\" (UID: \"71c4c37e-08cc-439c-9500-040c33d3fe94\") " pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.680854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.680827 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71c4c37e-08cc-439c-9500-040c33d3fe94-bound-sa-token\") pod \"cert-manager-55b4d954d5-ps8sw\" (UID: \"71c4c37e-08cc-439c-9500-040c33d3fe94\") " pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.681104 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.681083 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkw9w\" (UniqueName: \"kubernetes.io/projected/71c4c37e-08cc-439c-9500-040c33d3fe94-kube-api-access-lkw9w\") pod \"cert-manager-55b4d954d5-ps8sw\" (UID: \"71c4c37e-08cc-439c-9500-040c33d3fe94\") " pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.731879 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.731860 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ps8sw" Sep 08 18:44:05.999812 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:05.999786 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ps8sw"] Sep 08 18:44:06.002726 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:44:06.002702 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71c4c37e_08cc_439c_9500_040c33d3fe94.slice/crio-4167a0342b2717d7de7d3bf029c31ff48d8681f167a1223820f2f539224e24a1 WatchSource:0}: Error finding container 4167a0342b2717d7de7d3bf029c31ff48d8681f167a1223820f2f539224e24a1: Status 404 returned error can't find the container with id 4167a0342b2717d7de7d3bf029c31ff48d8681f167a1223820f2f539224e24a1 Sep 08 18:44:06.012077 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:06.012054 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ps8sw" event={"ID":"71c4c37e-08cc-439c-9500-040c33d3fe94","Type":"ContainerStarted","Data":"4167a0342b2717d7de7d3bf029c31ff48d8681f167a1223820f2f539224e24a1"} Sep 08 18:44:09.031140 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:09.031105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ps8sw" event={"ID":"71c4c37e-08cc-439c-9500-040c33d3fe94","Type":"ContainerStarted","Data":"dca42c34953ca21f0901909522de8c06c96d1501e387e6a4644b649861f10757"} Sep 08 18:44:09.047831 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:09.047789 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ps8sw" podStartSLOduration=1.23210892 podStartE2EDuration="4.047777113s" totalImagesPullingTime="2.815668193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:09.047014549 +0000 UTC m=+607.556166991" watchObservedRunningTime="2026-09-08 18:44:09.047777113 +0000 UTC m=+607.556929554" Sep 08 18:44:14.592265 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.592229 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d45bc8787-wwntb"] Sep 08 18:44:14.595353 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.595336 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.597347 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.597325 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 18:44:14.598259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.598241 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 18:44:14.598330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.598242 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 18:44:14.598330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.598293 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vd5bl\"" Sep 08 18:44:14.598562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.598540 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 18:44:14.598735 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.598692 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 18:44:14.602709 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.602692 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 18:44:14.606811 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.606787 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d45bc8787-wwntb"] Sep 08 18:44:14.631453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631432 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-oauth-config\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.631541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631465 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-config\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.631541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631486 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-serving-cert\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.631541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631519 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-service-ca\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.631637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631554 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stx9c\" (UniqueName: \"kubernetes.io/projected/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-kube-api-access-stx9c\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.631637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-trusted-ca-bundle\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.631637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.631596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-oauth-serving-cert\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732591 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732570 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-serving-cert\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732603 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-service-ca\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stx9c\" (UniqueName: \"kubernetes.io/projected/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-kube-api-access-stx9c\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732673 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-trusted-ca-bundle\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732831 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732702 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-oauth-serving-cert\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732831 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-oauth-config\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.732831 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.732760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-config\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.733330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.733307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-service-ca\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.733421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.733400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-oauth-serving-cert\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.733491 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.733459 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-config\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.733579 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.733557 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-trusted-ca-bundle\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.735063 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.735038 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-oauth-config\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.735183 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.735165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-console-serving-cert\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.739765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.739746 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stx9c\" (UniqueName: \"kubernetes.io/projected/dc3c0498-6137-4e96-aeda-81d6a8e2a1f9-kube-api-access-stx9c\") pod \"console-6d45bc8787-wwntb\" (UID: \"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9\") " pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:14.904095 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:14.904043 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:15.172816 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:15.172744 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d45bc8787-wwntb"] Sep 08 18:44:15.176351 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:44:15.176318 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3c0498_6137_4e96_aeda_81d6a8e2a1f9.slice/crio-fc4786528e508c891bf1e3f15b0c8455df30e8aaafaf86271a3c5685317942fc WatchSource:0}: Error finding container fc4786528e508c891bf1e3f15b0c8455df30e8aaafaf86271a3c5685317942fc: Status 404 returned error can't find the container with id fc4786528e508c891bf1e3f15b0c8455df30e8aaafaf86271a3c5685317942fc Sep 08 18:44:16.057331 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:16.057300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d45bc8787-wwntb" event={"ID":"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9","Type":"ContainerStarted","Data":"1033b488dc78850c0b46184bc2c6525fffdc2a46e95f346bca4040cc8972c669"} Sep 08 18:44:16.057702 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:16.057345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d45bc8787-wwntb" event={"ID":"dc3c0498-6137-4e96-aeda-81d6a8e2a1f9","Type":"ContainerStarted","Data":"fc4786528e508c891bf1e3f15b0c8455df30e8aaafaf86271a3c5685317942fc"} Sep 08 18:44:16.074218 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:16.074172 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d45bc8787-wwntb" podStartSLOduration=2.074158213 podStartE2EDuration="2.074158213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:16.073100745 +0000 UTC m=+614.582253186" watchObservedRunningTime="2026-09-08 18:44:16.074158213 +0000 UTC m=+614.583310653" Sep 08 18:44:20.871910 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.871880 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q"] Sep 08 18:44:20.875268 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.875253 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:20.877776 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.877750 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 18:44:20.877899 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.877848 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z6qp9\"" Sep 08 18:44:20.877899 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.877783 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 18:44:20.878035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.877912 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 18:44:20.878035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.877924 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 18:44:20.888618 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.888596 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q"] Sep 08 18:44:20.969834 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.969803 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:20.969960 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.969851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:20.969960 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.969905 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:20.969960 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.969930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:20.970095 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:20.969980 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctp7\" (UniqueName: \"kubernetes.io/projected/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-kube-api-access-dctp7\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.070974 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.070951 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctp7\" (UniqueName: \"kubernetes.io/projected/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-kube-api-access-dctp7\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.071083 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.070990 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.071083 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.071016 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.071083 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.071042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.071083 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.071067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.071439 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.071414 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.073229 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.073202 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.073492 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.073473 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.073565 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.073515 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.078630 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.078606 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctp7\" (UniqueName: \"kubernetes.io/projected/9e7ff63c-240f-4f78-9aba-7262ffa0aa13-kube-api-access-dctp7\") pod \"opendatahub-operator-controller-manager-67894858cf-vgx2q\" (UID: \"9e7ff63c-240f-4f78-9aba-7262ffa0aa13\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.185509 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.185462 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:21.460903 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.460834 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q"] Sep 08 18:44:21.465305 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:44:21.465274 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7ff63c_240f_4f78_9aba_7262ffa0aa13.slice/crio-9a03c4f0485e3411a85ed873bf29f314b45976639a164253e04d90093ed9e0cd WatchSource:0}: Error finding container 9a03c4f0485e3411a85ed873bf29f314b45976639a164253e04d90093ed9e0cd: Status 404 returned error can't find the container with id 9a03c4f0485e3411a85ed873bf29f314b45976639a164253e04d90093ed9e0cd Sep 08 18:44:21.467793 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:21.467774 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:44:22.081672 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:22.081625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" event={"ID":"9e7ff63c-240f-4f78-9aba-7262ffa0aa13","Type":"ContainerStarted","Data":"9a03c4f0485e3411a85ed873bf29f314b45976639a164253e04d90093ed9e0cd"} Sep 08 18:44:24.904400 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:24.904334 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:24.904796 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:24.904593 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:24.909235 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:24.909217 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:25.097421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:25.097393 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" event={"ID":"9e7ff63c-240f-4f78-9aba-7262ffa0aa13","Type":"ContainerStarted","Data":"b9007ba8bf9085d54935e638282ef25e9a790b87b64fd7cb10dd6f423c9e27b3"} Sep 08 18:44:25.101073 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:25.101052 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d45bc8787-wwntb" Sep 08 18:44:26.102853 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:26.102818 2557 generic.go:356] "Generic (PLEG): container finished" podID="9e7ff63c-240f-4f78-9aba-7262ffa0aa13" containerID="b9007ba8bf9085d54935e638282ef25e9a790b87b64fd7cb10dd6f423c9e27b3" exitCode=0 Sep 08 18:44:26.103281 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:26.102887 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" event={"ID":"9e7ff63c-240f-4f78-9aba-7262ffa0aa13","Type":"ContainerDied","Data":"b9007ba8bf9085d54935e638282ef25e9a790b87b64fd7cb10dd6f423c9e27b3"} Sep 08 18:44:27.108785 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:27.108750 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" event={"ID":"9e7ff63c-240f-4f78-9aba-7262ffa0aa13","Type":"ContainerStarted","Data":"d7777087a137b09ffd5ad0ffaf15cd8fc1fb40e65f2a847cbef230a09a93aa5a"} Sep 08 18:44:27.128670 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:27.128608 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" podStartSLOduration=2.688445415 podStartE2EDuration="7.128593875s" totalImagesPullingTime="3.43186232s" totalInitContainerRuntime="1.00828614s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:27.127558003 +0000 UTC m=+625.636710435" watchObservedRunningTime="2026-09-08 18:44:27.128593875 +0000 UTC m=+625.637746315" Sep 08 18:44:28.113135 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:28.113106 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:39.118909 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:39.118875 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-vgx2q" Sep 08 18:44:41.110719 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.110688 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54b8878898-mdqxh"] Sep 08 18:44:41.119556 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.119534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.122674 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.122630 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 18:44:41.122861 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.122844 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 18:44:41.122930 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.122914 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 18:44:41.123630 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.123611 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gtwxl\"" Sep 08 18:44:41.123726 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.123640 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 18:44:41.131282 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.131260 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54b8878898-mdqxh"] Sep 08 18:44:41.205096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.205066 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5453970-29ad-44f4-8bee-1bd0c7e87666-tls-certs\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.205216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.205098 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5453970-29ad-44f4-8bee-1bd0c7e87666-tmp\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.205216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.205191 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqs9\" (UniqueName: \"kubernetes.io/projected/c5453970-29ad-44f4-8bee-1bd0c7e87666-kube-api-access-drqs9\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.305986 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.305954 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqs9\" (UniqueName: \"kubernetes.io/projected/c5453970-29ad-44f4-8bee-1bd0c7e87666-kube-api-access-drqs9\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.306085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.305991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5453970-29ad-44f4-8bee-1bd0c7e87666-tls-certs\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.306085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.306022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5453970-29ad-44f4-8bee-1bd0c7e87666-tmp\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.306190 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:44:41.306097 2557 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 08 18:44:41.306190 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:44:41.306143 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5453970-29ad-44f4-8bee-1bd0c7e87666-tls-certs podName:c5453970-29ad-44f4-8bee-1bd0c7e87666 nodeName:}" failed. No retries permitted until 2026-09-08 18:44:41.806129262 +0000 UTC m=+640.315281682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c5453970-29ad-44f4-8bee-1bd0c7e87666-tls-certs") pod "kube-auth-proxy-54b8878898-mdqxh" (UID: "c5453970-29ad-44f4-8bee-1bd0c7e87666") : secret "kube-auth-proxy-tls" not found Sep 08 18:44:41.308108 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.308092 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5453970-29ad-44f4-8bee-1bd0c7e87666-tmp\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.314516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.314495 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqs9\" (UniqueName: \"kubernetes.io/projected/c5453970-29ad-44f4-8bee-1bd0c7e87666-kube-api-access-drqs9\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.809545 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.809514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5453970-29ad-44f4-8bee-1bd0c7e87666-tls-certs\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:41.811785 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:41.811767 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5453970-29ad-44f4-8bee-1bd0c7e87666-tls-certs\") pod \"kube-auth-proxy-54b8878898-mdqxh\" (UID: \"c5453970-29ad-44f4-8bee-1bd0c7e87666\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:42.028555 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.028529 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" Sep 08 18:44:42.304422 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.304364 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54b8878898-mdqxh"] Sep 08 18:44:42.306800 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:44:42.306773 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5453970_29ad_44f4_8bee_1bd0c7e87666.slice/crio-f077817d6e4c82bb8b90b03b7d441a0e3a87f994170dfef505bc5eb93248d491 WatchSource:0}: Error finding container f077817d6e4c82bb8b90b03b7d441a0e3a87f994170dfef505bc5eb93248d491: Status 404 returned error can't find the container with id f077817d6e4c82bb8b90b03b7d441a0e3a87f994170dfef505bc5eb93248d491 Sep 08 18:44:42.377710 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.377621 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf"] Sep 08 18:44:42.385482 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.385461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.387863 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.387825 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 18:44:42.387972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.387922 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 18:44:42.387972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.387933 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 18:44:42.387972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.387944 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tpvnd\"" Sep 08 18:44:42.387972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.387931 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 18:44:42.393691 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.393668 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf"] Sep 08 18:44:42.414342 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jpz\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-kube-api-access-69jpz\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.414445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414356 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.414445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414422 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.414555 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414461 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.414555 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414483 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.414555 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414512 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.414555 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.414541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515017 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.514993 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jpz\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-kube-api-access-69jpz\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515048 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515069 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515198 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.515652 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.515626 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.517336 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.517307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.517336 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.517332 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.517480 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.517343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.517480 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.517451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.522541 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.522523 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jpz\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-kube-api-access-69jpz\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.522662 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.522561 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-txxhf\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.695265 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.695214 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:42.966373 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:42.966290 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf"] Sep 08 18:44:42.969828 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:44:42.969794 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad5bc1b_aa5e_463a_885b_25c7ac60ffde.slice/crio-c3a8fa4aaec09f83cb3d6d26a9564bc6bc488f7cc817463087c62fb6ab082adc WatchSource:0}: Error finding container c3a8fa4aaec09f83cb3d6d26a9564bc6bc488f7cc817463087c62fb6ab082adc: Status 404 returned error can't find the container with id c3a8fa4aaec09f83cb3d6d26a9564bc6bc488f7cc817463087c62fb6ab082adc Sep 08 18:44:43.176540 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:43.176494 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" event={"ID":"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde","Type":"ContainerStarted","Data":"c3a8fa4aaec09f83cb3d6d26a9564bc6bc488f7cc817463087c62fb6ab082adc"} Sep 08 18:44:43.178055 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:43.178016 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" event={"ID":"c5453970-29ad-44f4-8bee-1bd0c7e87666","Type":"ContainerStarted","Data":"f077817d6e4c82bb8b90b03b7d441a0e3a87f994170dfef505bc5eb93248d491"} Sep 08 18:44:46.194953 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:46.194919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" event={"ID":"c5453970-29ad-44f4-8bee-1bd0c7e87666","Type":"ContainerStarted","Data":"d606ed65af8e12c9d90f00667b605a3a02bb9d28568f17de0445f3145a069a97"} Sep 08 18:44:46.211255 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:46.211208 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54b8878898-mdqxh" podStartSLOduration=2.29548227 podStartE2EDuration="5.211192601s" totalImagesPullingTime="2.915710331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:46.209893106 +0000 UTC m=+644.719045547" watchObservedRunningTime="2026-09-08 18:44:46.211192601 +0000 UTC m=+644.720345041" Sep 08 18:44:51.962703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:51.962667 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:44:51.962983 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:51.962737 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:44:52.224730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:52.224694 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" event={"ID":"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde","Type":"ContainerStarted","Data":"21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e"} Sep 08 18:44:52.224919 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:52.224856 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:53.230468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:53.230442 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:44:53.248349 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:44:53.248303 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" podStartSLOduration=2.258357041 podStartE2EDuration="11.248287322s" totalImagesPullingTime="8.989930281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:52.264991821 +0000 UTC m=+650.774144263" watchObservedRunningTime="2026-09-08 18:44:53.248287322 +0000 UTC m=+651.757439763" Sep 08 18:45:02.369110 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.369073 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mplfc"] Sep 08 18:45:02.374198 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.374173 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.382510 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.382492 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 08 18:45:02.382926 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.382902 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cbgnh\"" Sep 08 18:45:02.397723 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.397699 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z"] Sep 08 18:45:02.401157 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.401137 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mplfc"] Sep 08 18:45:02.401242 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.401231 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:02.405093 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.405075 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 08 18:45:02.405427 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.405412 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fsmzd\"" Sep 08 18:45:02.425344 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.425319 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z"] Sep 08 18:45:02.462184 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.462155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfrz\" (UniqueName: \"kubernetes.io/projected/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-kube-api-access-6sfrz\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:02.462287 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.462192 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3072186-cf89-4e75-a8ee-1dc8645b4797-cert\") pod \"kserve-controller-manager-d6f4b65b8-mplfc\" (UID: \"e3072186-cf89-4e75-a8ee-1dc8645b4797\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.462287 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.462216 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:02.462372 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.462290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4rl\" (UniqueName: \"kubernetes.io/projected/e3072186-cf89-4e75-a8ee-1dc8645b4797-kube-api-access-fx4rl\") pod \"kserve-controller-manager-d6f4b65b8-mplfc\" (UID: \"e3072186-cf89-4e75-a8ee-1dc8645b4797\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.562821 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.562784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3072186-cf89-4e75-a8ee-1dc8645b4797-cert\") pod \"kserve-controller-manager-d6f4b65b8-mplfc\" (UID: \"e3072186-cf89-4e75-a8ee-1dc8645b4797\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.562936 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.562828 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:02.562936 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.562859 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4rl\" (UniqueName: \"kubernetes.io/projected/e3072186-cf89-4e75-a8ee-1dc8645b4797-kube-api-access-fx4rl\") pod \"kserve-controller-manager-d6f4b65b8-mplfc\" (UID: \"e3072186-cf89-4e75-a8ee-1dc8645b4797\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.563045 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:45:02.562930 2557 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 08 18:45:02.563045 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.562971 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfrz\" (UniqueName: \"kubernetes.io/projected/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-kube-api-access-6sfrz\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:02.563045 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:45:02.562991 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-cert podName:62aab1fa-b2e9-43e4-ac83-a0edabd10f8f nodeName:}" failed. No retries permitted until 2026-09-08 18:45:03.062971793 +0000 UTC m=+661.572124214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-cert") pod "llmisvc-controller-manager-777c6bd7d5-wqs8z" (UID: "62aab1fa-b2e9-43e4-ac83-a0edabd10f8f") : secret "llmisvc-webhook-server-cert" not found Sep 08 18:45:02.565080 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.565061 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3072186-cf89-4e75-a8ee-1dc8645b4797-cert\") pod \"kserve-controller-manager-d6f4b65b8-mplfc\" (UID: \"e3072186-cf89-4e75-a8ee-1dc8645b4797\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.572529 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.572511 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4rl\" (UniqueName: \"kubernetes.io/projected/e3072186-cf89-4e75-a8ee-1dc8645b4797-kube-api-access-fx4rl\") pod \"kserve-controller-manager-d6f4b65b8-mplfc\" (UID: \"e3072186-cf89-4e75-a8ee-1dc8645b4797\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.578793 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.578773 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfrz\" (UniqueName: \"kubernetes.io/projected/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-kube-api-access-6sfrz\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:02.684962 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.684896 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:02.954131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:02.954056 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mplfc"] Sep 08 18:45:02.957343 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:45:02.957318 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3072186_cf89_4e75_a8ee_1dc8645b4797.slice/crio-8b03beff94422621352dc8886b0a901156f70d43d4ca579cc617fc4854dd22ef WatchSource:0}: Error finding container 8b03beff94422621352dc8886b0a901156f70d43d4ca579cc617fc4854dd22ef: Status 404 returned error can't find the container with id 8b03beff94422621352dc8886b0a901156f70d43d4ca579cc617fc4854dd22ef Sep 08 18:45:03.067637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:03.067608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:03.069779 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:03.069762 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62aab1fa-b2e9-43e4-ac83-a0edabd10f8f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqs8z\" (UID: \"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:03.270904 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:03.270869 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" event={"ID":"e3072186-cf89-4e75-a8ee-1dc8645b4797","Type":"ContainerStarted","Data":"8b03beff94422621352dc8886b0a901156f70d43d4ca579cc617fc4854dd22ef"} Sep 08 18:45:03.313261 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:03.313239 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:03.591727 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:03.591700 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z"] Sep 08 18:45:03.593582 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:45:03.593554 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod62aab1fa_b2e9_43e4_ac83_a0edabd10f8f.slice/crio-b5772ab57a032dd8d7e3f65c909bc3bdc44a232eecdd5d3788905bb77ef73747 WatchSource:0}: Error finding container b5772ab57a032dd8d7e3f65c909bc3bdc44a232eecdd5d3788905bb77ef73747: Status 404 returned error can't find the container with id b5772ab57a032dd8d7e3f65c909bc3bdc44a232eecdd5d3788905bb77ef73747 Sep 08 18:45:04.189162 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.189127 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf"] Sep 08 18:45:04.189447 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.189417 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" podUID="5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" containerName="discovery" containerID="cri-o://21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e" gracePeriod=30 Sep 08 18:45:04.277344 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.277313 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" event={"ID":"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f","Type":"ContainerStarted","Data":"b5772ab57a032dd8d7e3f65c909bc3bdc44a232eecdd5d3788905bb77ef73747"} Sep 08 18:45:04.455345 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.455317 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:45:04.579257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579227 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-dns-cert\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.579420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579266 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-token\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.579420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579284 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-cacerts\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.579420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579325 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69jpz\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-kube-api-access-69jpz\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.579420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579349 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-local-certs\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.579629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-ca-configmap\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.579629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.579556 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-kubeconfig\") pod \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\" (UID: \"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde\") " Sep 08 18:45:04.580314 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.580115 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:45:04.582293 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.582265 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:45:04.582510 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.582416 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:45:04.582594 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.582539 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:45:04.582594 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.582557 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-cacerts" (OuterVolumeSpecName: "cacerts") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:45:04.582704 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.582604 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-local-certs" (OuterVolumeSpecName: "local-certs") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:45:04.583817 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.583793 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-kube-api-access-69jpz" (OuterVolumeSpecName: "kube-api-access-69jpz") pod "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" (UID: "5ad5bc1b-aa5e-463a-885b-25c7ac60ffde"). InnerVolumeSpecName "kube-api-access-69jpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:45:04.680814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680780 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-dns-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:04.680814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680808 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:04.680814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680816 2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-cacerts\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:04.681195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680824 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jpz\" (UniqueName: \"kubernetes.io/projected/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-kube-api-access-69jpz\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:04.681195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680833 2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-local-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:04.681195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680841 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-csr-ca-configmap\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:04.681195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:04.680851 2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde-istio-kubeconfig\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:45:05.285403 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.285363 2557 generic.go:356] "Generic (PLEG): container finished" podID="5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" containerID="21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e" exitCode=0 Sep 08 18:45:05.285571 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.285421 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" Sep 08 18:45:05.285571 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.285430 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" event={"ID":"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde","Type":"ContainerDied","Data":"21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e"} Sep 08 18:45:05.285571 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.285482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf" event={"ID":"5ad5bc1b-aa5e-463a-885b-25c7ac60ffde","Type":"ContainerDied","Data":"c3a8fa4aaec09f83cb3d6d26a9564bc6bc488f7cc817463087c62fb6ab082adc"} Sep 08 18:45:05.285571 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.285508 2557 scope.go:117] "RemoveContainer" containerID="21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e" Sep 08 18:45:05.313290 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.313255 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf"] Sep 08 18:45:05.317124 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.317097 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txxhf"] Sep 08 18:45:05.633907 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.633889 2557 scope.go:117] "RemoveContainer" containerID="21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e" Sep 08 18:45:05.634267 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:45:05.634246 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e\": container with ID starting with 21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e not found: ID does not exist" containerID="21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e" Sep 08 18:45:05.634318 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:05.634277 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e"} err="failed to get container status \"21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e\": rpc error: code = NotFound desc = could not find container \"21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e\": container with ID starting with 21ec7864d47807aab07e5d56ccc44cf4d84752fba26604d3f793fac40ad7e04e not found: ID does not exist" Sep 08 18:45:06.025437 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:06.025401 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" path="/var/lib/kubelet/pods/5ad5bc1b-aa5e-463a-885b-25c7ac60ffde/volumes" Sep 08 18:45:06.291481 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:06.291402 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" event={"ID":"e3072186-cf89-4e75-a8ee-1dc8645b4797","Type":"ContainerStarted","Data":"eb178786c3e0ebae6ff7403b1f99c488453e804c8d10ecf0aaffdc57f57a4b27"} Sep 08 18:45:06.291606 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:06.291537 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:06.306047 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:06.305996 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" podStartSLOduration=1.584530672 podStartE2EDuration="4.305971829s" totalImagesPullingTime="2.721441157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:45:06.304514791 +0000 UTC m=+664.813667231" watchObservedRunningTime="2026-09-08 18:45:06.305971829 +0000 UTC m=+664.815124270" Sep 08 18:45:07.297093 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:07.297052 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" event={"ID":"62aab1fa-b2e9-43e4-ac83-a0edabd10f8f","Type":"ContainerStarted","Data":"387b277e27038b7bc189adc28d696932bf0cd6504e095ccf4887d5e4a6034d43"} Sep 08 18:45:07.311325 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:07.311282 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" podStartSLOduration=1.7601476539999998 podStartE2EDuration="5.311269701s" totalImagesPullingTime="3.551122047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:45:07.311114059 +0000 UTC m=+665.820266499" watchObservedRunningTime="2026-09-08 18:45:07.311269701 +0000 UTC m=+665.820422191" Sep 08 18:45:08.301279 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:08.301249 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:45:37.301527 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:37.301499 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mplfc" Sep 08 18:45:39.321037 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:45:39.321007 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqs8z" Sep 08 18:46:08.628631 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.628599 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2nmgn"] Sep 08 18:46:08.629057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.628911 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" containerName="discovery" Sep 08 18:46:08.629057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.628922 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" containerName="discovery" Sep 08 18:46:08.629057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.628988 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad5bc1b-aa5e-463a-885b-25c7ac60ffde" containerName="discovery" Sep 08 18:46:08.633065 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.633049 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:08.635429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.635400 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 18:46:08.635551 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.635517 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 18:46:08.635551 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.635529 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xf6bg\"" Sep 08 18:46:08.640729 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.640708 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2nmgn"] Sep 08 18:46:08.723018 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.722989 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r865x\" (UniqueName: \"kubernetes.io/projected/f24b6e59-e45f-41ae-a466-f57ed7d90134-kube-api-access-r865x\") pod \"authorino-operator-76b478bd78-2nmgn\" (UID: \"f24b6e59-e45f-41ae-a466-f57ed7d90134\") " pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:08.824014 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.823989 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r865x\" (UniqueName: \"kubernetes.io/projected/f24b6e59-e45f-41ae-a466-f57ed7d90134-kube-api-access-r865x\") pod \"authorino-operator-76b478bd78-2nmgn\" (UID: \"f24b6e59-e45f-41ae-a466-f57ed7d90134\") " pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:08.834956 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.834932 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r865x\" (UniqueName: \"kubernetes.io/projected/f24b6e59-e45f-41ae-a466-f57ed7d90134-kube-api-access-r865x\") pod \"authorino-operator-76b478bd78-2nmgn\" (UID: \"f24b6e59-e45f-41ae-a466-f57ed7d90134\") " pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:08.943249 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:08.943181 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:09.225776 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:09.225747 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2nmgn"] Sep 08 18:46:09.229066 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:09.229034 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24b6e59_e45f_41ae_a466_f57ed7d90134.slice/crio-17adf975761b3957d2ed1e43edc13921112194e2f4d72be011bd3aa2828d98e3 WatchSource:0}: Error finding container 17adf975761b3957d2ed1e43edc13921112194e2f4d72be011bd3aa2828d98e3: Status 404 returned error can't find the container with id 17adf975761b3957d2ed1e43edc13921112194e2f4d72be011bd3aa2828d98e3 Sep 08 18:46:09.557401 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:09.557322 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" event={"ID":"f24b6e59-e45f-41ae-a466-f57ed7d90134","Type":"ContainerStarted","Data":"17adf975761b3957d2ed1e43edc13921112194e2f4d72be011bd3aa2828d98e3"} Sep 08 18:46:10.821656 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:10.821622 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd"] Sep 08 18:46:10.825127 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:10.825106 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:10.827533 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:10.827506 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tgw7n\"" Sep 08 18:46:10.835879 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:10.835860 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd"] Sep 08 18:46:10.938632 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:10.938602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhkj\" (UniqueName: \"kubernetes.io/projected/e328bde8-a577-4594-8b23-79a5a4202b17-kube-api-access-dnhkj\") pod \"limitador-operator-controller-manager-74495c69f8-r2zbd\" (UID: \"e328bde8-a577-4594-8b23-79a5a4202b17\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:11.039079 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:11.039039 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhkj\" (UniqueName: \"kubernetes.io/projected/e328bde8-a577-4594-8b23-79a5a4202b17-kube-api-access-dnhkj\") pod \"limitador-operator-controller-manager-74495c69f8-r2zbd\" (UID: \"e328bde8-a577-4594-8b23-79a5a4202b17\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:11.063428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:11.063405 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhkj\" (UniqueName: \"kubernetes.io/projected/e328bde8-a577-4594-8b23-79a5a4202b17-kube-api-access-dnhkj\") pod \"limitador-operator-controller-manager-74495c69f8-r2zbd\" (UID: \"e328bde8-a577-4594-8b23-79a5a4202b17\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:11.139153 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:11.139090 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:12.322583 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:12.322497 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd"] Sep 08 18:46:12.325261 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:12.325233 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode328bde8_a577_4594_8b23_79a5a4202b17.slice/crio-267d512ecfb490a93b1a47b66c5244ba9684705df5322582f60a056e9dfc8cba WatchSource:0}: Error finding container 267d512ecfb490a93b1a47b66c5244ba9684705df5322582f60a056e9dfc8cba: Status 404 returned error can't find the container with id 267d512ecfb490a93b1a47b66c5244ba9684705df5322582f60a056e9dfc8cba Sep 08 18:46:12.574770 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:12.574692 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" event={"ID":"e328bde8-a577-4594-8b23-79a5a4202b17","Type":"ContainerStarted","Data":"267d512ecfb490a93b1a47b66c5244ba9684705df5322582f60a056e9dfc8cba"} Sep 08 18:46:12.576230 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:12.576198 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" event={"ID":"f24b6e59-e45f-41ae-a466-f57ed7d90134","Type":"ContainerStarted","Data":"adadc2a548b1133d3bac40c2c6355bab3633e737031dc03b041dfa33c3896019"} Sep 08 18:46:12.576347 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:12.576334 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:12.593332 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:12.593290 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" podStartSLOduration=1.697696131 podStartE2EDuration="4.593277336s" totalImagesPullingTime="2.895581205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:12.591940022 +0000 UTC m=+731.101092465" watchObservedRunningTime="2026-09-08 18:46:12.593277336 +0000 UTC m=+731.102429776" Sep 08 18:46:13.112792 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.112765 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8"] Sep 08 18:46:13.116204 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.116190 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:13.118436 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.118413 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 08 18:46:13.118576 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.118421 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kbd9m\"" Sep 08 18:46:13.125818 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.125797 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8"] Sep 08 18:46:13.154822 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.154797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhxv\" (UniqueName: \"kubernetes.io/projected/b48cadc3-3fa5-44a4-9ed3-964628f12f3d-kube-api-access-bzhxv\") pod \"dns-operator-controller-manager-7b84cd7c9c-shwt8\" (UID: \"b48cadc3-3fa5-44a4-9ed3-964628f12f3d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:13.255978 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.255941 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhxv\" (UniqueName: \"kubernetes.io/projected/b48cadc3-3fa5-44a4-9ed3-964628f12f3d-kube-api-access-bzhxv\") pod \"dns-operator-controller-manager-7b84cd7c9c-shwt8\" (UID: \"b48cadc3-3fa5-44a4-9ed3-964628f12f3d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:13.267515 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.267491 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhxv\" (UniqueName: \"kubernetes.io/projected/b48cadc3-3fa5-44a4-9ed3-964628f12f3d-kube-api-access-bzhxv\") pod \"dns-operator-controller-manager-7b84cd7c9c-shwt8\" (UID: \"b48cadc3-3fa5-44a4-9ed3-964628f12f3d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:13.427418 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.427367 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:13.704698 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:13.704479 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8"] Sep 08 18:46:13.707387 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:13.707363 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48cadc3_3fa5_44a4_9ed3_964628f12f3d.slice/crio-ad2cb9e468f602b9504c0256a7b1abecc4e6b12294da2293d2e14edf75f287d9 WatchSource:0}: Error finding container ad2cb9e468f602b9504c0256a7b1abecc4e6b12294da2293d2e14edf75f287d9: Status 404 returned error can't find the container with id ad2cb9e468f602b9504c0256a7b1abecc4e6b12294da2293d2e14edf75f287d9 Sep 08 18:46:14.586468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:14.586435 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" event={"ID":"b48cadc3-3fa5-44a4-9ed3-964628f12f3d","Type":"ContainerStarted","Data":"ad2cb9e468f602b9504c0256a7b1abecc4e6b12294da2293d2e14edf75f287d9"} Sep 08 18:46:15.595319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:15.595293 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" event={"ID":"e328bde8-a577-4594-8b23-79a5a4202b17","Type":"ContainerStarted","Data":"b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b"} Sep 08 18:46:15.595705 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:15.595384 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:16.601873 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:16.601840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" event={"ID":"b48cadc3-3fa5-44a4-9ed3-964628f12f3d","Type":"ContainerStarted","Data":"6c7136c7adfbdd67505ad5690fed362f0747c081ea5a3517963e01ba057885b9"} Sep 08 18:46:16.602239 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:16.602003 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:16.626824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:16.626781 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" podStartSLOduration=1.756511129 podStartE2EDuration="3.626769318s" totalImagesPullingTime="1.870258189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:16.625633327 +0000 UTC m=+735.134785768" watchObservedRunningTime="2026-09-08 18:46:16.626769318 +0000 UTC m=+735.135921758" Sep 08 18:46:16.627201 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:16.627181 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" podStartSLOduration=4.331271781 podStartE2EDuration="6.627175755s" totalImagesPullingTime="2.295903974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:15.614877344 +0000 UTC m=+734.124029785" watchObservedRunningTime="2026-09-08 18:46:16.627175755 +0000 UTC m=+735.136328203" Sep 08 18:46:17.227876 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.227847 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf"] Sep 08 18:46:17.231155 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.231137 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.233355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.233337 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k6xgw\"" Sep 08 18:46:17.246695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.246671 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf"] Sep 08 18:46:17.289947 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.289922 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbs6\" (UniqueName: \"kubernetes.io/projected/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-kube-api-access-rrbs6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.290032 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.289958 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.391015 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.390991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.391096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.391051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbs6\" (UniqueName: \"kubernetes.io/projected/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-kube-api-access-rrbs6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.391362 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.391342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.398027 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.398001 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbs6\" (UniqueName: \"kubernetes.io/projected/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-kube-api-access-rrbs6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.541185 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.541130 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:17.814553 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:17.814443 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf"] Sep 08 18:46:17.817258 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:17.817232 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b8aacf_bf59_4c8c_b7a9_0f4d8a623610.slice/crio-cdd72b02a9b54395f31ca4e24dcafd123e6380ad0d60e3718dd7c6038462f35c WatchSource:0}: Error finding container cdd72b02a9b54395f31ca4e24dcafd123e6380ad0d60e3718dd7c6038462f35c: Status 404 returned error can't find the container with id cdd72b02a9b54395f31ca4e24dcafd123e6380ad0d60e3718dd7c6038462f35c Sep 08 18:46:18.615291 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:18.615258 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" event={"ID":"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610","Type":"ContainerStarted","Data":"cdd72b02a9b54395f31ca4e24dcafd123e6380ad0d60e3718dd7c6038462f35c"} Sep 08 18:46:22.639458 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:22.639430 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" event={"ID":"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610","Type":"ContainerStarted","Data":"be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4"} Sep 08 18:46:22.639834 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:22.639542 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:22.658989 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:22.658940 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" podStartSLOduration=0.999680344 podStartE2EDuration="5.658923161s" totalImagesPullingTime="4.659242817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:22.657459057 +0000 UTC m=+741.166611497" watchObservedRunningTime="2026-09-08 18:46:22.658923161 +0000 UTC m=+741.168075603" Sep 08 18:46:23.582778 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:23.582750 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-2nmgn" Sep 08 18:46:26.604209 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:26.604181 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:27.611782 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:27.611750 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-shwt8" Sep 08 18:46:33.646953 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:33.646922 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:34.538605 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.538571 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g"] Sep 08 18:46:34.541170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.541155 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.555897 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.555877 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g"] Sep 08 18:46:34.621524 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.621495 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1866a50d-1960-4959-912a-301e8cb2805d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.621668 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.621536 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbhv\" (UniqueName: \"kubernetes.io/projected/1866a50d-1960-4959-912a-301e8cb2805d-kube-api-access-dcbhv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.722219 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.722198 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1866a50d-1960-4959-912a-301e8cb2805d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.722526 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.722229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbhv\" (UniqueName: \"kubernetes.io/projected/1866a50d-1960-4959-912a-301e8cb2805d-kube-api-access-dcbhv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.722567 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.722525 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1866a50d-1960-4959-912a-301e8cb2805d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.731307 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.731281 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbhv\" (UniqueName: \"kubernetes.io/projected/1866a50d-1960-4959-912a-301e8cb2805d-kube-api-access-dcbhv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:34.851151 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:34.851103 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:35.129139 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.129058 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g"] Sep 08 18:46:35.134052 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:35.134025 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1866a50d_1960_4959_912a_301e8cb2805d.slice/crio-78115574a12c6e9813c9c1e0a106e930d7bb140daa3b021fe3b50f299cbb8c34 WatchSource:0}: Error finding container 78115574a12c6e9813c9c1e0a106e930d7bb140daa3b021fe3b50f299cbb8c34: Status 404 returned error can't find the container with id 78115574a12c6e9813c9c1e0a106e930d7bb140daa3b021fe3b50f299cbb8c34 Sep 08 18:46:35.701301 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.701268 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" event={"ID":"1866a50d-1960-4959-912a-301e8cb2805d","Type":"ContainerStarted","Data":"967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e"} Sep 08 18:46:35.701496 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.701326 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" event={"ID":"1866a50d-1960-4959-912a-301e8cb2805d","Type":"ContainerStarted","Data":"78115574a12c6e9813c9c1e0a106e930d7bb140daa3b021fe3b50f299cbb8c34"} Sep 08 18:46:35.701496 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.701349 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:35.725330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.725283 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" podStartSLOduration=1.725266366 podStartE2EDuration="1.725266366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:35.724785648 +0000 UTC m=+754.233938088" watchObservedRunningTime="2026-09-08 18:46:35.725266366 +0000 UTC m=+754.234418807" Sep 08 18:46:35.765252 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.765224 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g"] Sep 08 18:46:35.774694 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.774673 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g"] Sep 08 18:46:35.787763 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.787744 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf"] Sep 08 18:46:35.787910 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.787889 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" containerName="manager" containerID="cri-o://be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4" gracePeriod=2 Sep 08 18:46:35.792205 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.792180 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw"] Sep 08 18:46:35.792706 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.792684 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1866a50d-1960-4959-912a-301e8cb2805d" containerName="manager" Sep 08 18:46:35.792706 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.792707 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1866a50d-1960-4959-912a-301e8cb2805d" containerName="manager" Sep 08 18:46:35.792861 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.792813 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1866a50d-1960-4959-912a-301e8cb2805d" containerName="manager" Sep 08 18:46:35.795126 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.795102 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:35.803446 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.803425 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf"] Sep 08 18:46:35.810479 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.810455 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw"] Sep 08 18:46:35.816172 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:35.816133 2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\": pods \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:35.819352 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.819330 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q"] Sep 08 18:46:35.820035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.820017 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" containerName="manager" Sep 08 18:46:35.820129 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.820089 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" containerName="manager" Sep 08 18:46:35.820250 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.820199 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" containerName="manager" Sep 08 18:46:35.822532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.822510 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:35.823144 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.823120 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd"] Sep 08 18:46:35.823314 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.823293 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" containerName="manager" containerID="cri-o://b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b" gracePeriod=2 Sep 08 18:46:35.837311 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.837277 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd"] Sep 08 18:46:35.840330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.840307 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q"] Sep 08 18:46:35.844978 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:35.844789 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:35.846764 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:35.846734 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r2zbd\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:35.848803 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:35.848781 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:35.853130 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.853111 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9"] Sep 08 18:46:35.853416 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.853405 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" containerName="manager" Sep 08 18:46:35.853458 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.853418 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" containerName="manager" Sep 08 18:46:35.853491 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.853465 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" containerName="manager" Sep 08 18:46:35.854982 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.854967 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:35.868914 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.868893 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9"] Sep 08 18:46:35.893159 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:35.893125 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r2zbd\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:35.894923 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:35.894898 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zrfwf\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:35.930714 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.930689 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbs8\" (UniqueName: \"kubernetes.io/projected/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-kube-api-access-qqbs8\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8fxpw\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:35.930814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.930752 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8fxpw\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:35.930814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.930783 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkpg\" (UniqueName: \"kubernetes.io/projected/3f508a96-5d80-429c-bbc6-c9f1a97560ba-kube-api-access-jgkpg\") pod \"limitador-operator-controller-manager-74495c69f8-fkrd9\" (UID: \"3f508a96-5d80-429c-bbc6-c9f1a97560ba\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:35.930927 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.930812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w472n\" (UniqueName: \"kubernetes.io/projected/5385e112-a64a-4cce-b31a-5a8f307a620e-kube-api-access-w472n\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8lh7q\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:35.930980 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:35.930939 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5385e112-a64a-4cce-b31a-5a8f307a620e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8lh7q\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.032312 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.032202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbs8\" (UniqueName: \"kubernetes.io/projected/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-kube-api-access-qqbs8\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8fxpw\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:36.032312 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.032288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8fxpw\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:36.032312 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.032324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkpg\" (UniqueName: \"kubernetes.io/projected/3f508a96-5d80-429c-bbc6-c9f1a97560ba-kube-api-access-jgkpg\") pod \"limitador-operator-controller-manager-74495c69f8-fkrd9\" (UID: \"3f508a96-5d80-429c-bbc6-c9f1a97560ba\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:36.032714 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.032359 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w472n\" (UniqueName: \"kubernetes.io/projected/5385e112-a64a-4cce-b31a-5a8f307a620e-kube-api-access-w472n\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8lh7q\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.032714 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.032411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5385e112-a64a-4cce-b31a-5a8f307a620e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8lh7q\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.033308 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.032872 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5385e112-a64a-4cce-b31a-5a8f307a620e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8lh7q\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.033308 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.033214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8fxpw\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:36.041586 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.041561 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbs8\" (UniqueName: \"kubernetes.io/projected/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-kube-api-access-qqbs8\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8fxpw\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:36.041719 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.041598 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkpg\" (UniqueName: \"kubernetes.io/projected/3f508a96-5d80-429c-bbc6-c9f1a97560ba-kube-api-access-jgkpg\") pod \"limitador-operator-controller-manager-74495c69f8-fkrd9\" (UID: \"3f508a96-5d80-429c-bbc6-c9f1a97560ba\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:36.041719 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.041608 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w472n\" (UniqueName: \"kubernetes.io/projected/5385e112-a64a-4cce-b31a-5a8f307a620e-kube-api-access-w472n\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8lh7q\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.060533 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.060514 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:36.062759 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.062740 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:36.133431 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.133404 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-extensions-socket-volume\") pod \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " Sep 08 18:46:36.133562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.133444 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrbs6\" (UniqueName: \"kubernetes.io/projected/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-kube-api-access-rrbs6\") pod \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\" (UID: \"d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610\") " Sep 08 18:46:36.133562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.133510 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnhkj\" (UniqueName: \"kubernetes.io/projected/e328bde8-a577-4594-8b23-79a5a4202b17-kube-api-access-dnhkj\") pod \"e328bde8-a577-4594-8b23-79a5a4202b17\" (UID: \"e328bde8-a577-4594-8b23-79a5a4202b17\") " Sep 08 18:46:36.133915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.133894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" (UID: "d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:36.135605 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.135579 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-kube-api-access-rrbs6" (OuterVolumeSpecName: "kube-api-access-rrbs6") pod "d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" (UID: "d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610"). InnerVolumeSpecName "kube-api-access-rrbs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:36.135698 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.135637 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e328bde8-a577-4594-8b23-79a5a4202b17-kube-api-access-dnhkj" (OuterVolumeSpecName: "kube-api-access-dnhkj") pod "e328bde8-a577-4594-8b23-79a5a4202b17" (UID: "e328bde8-a577-4594-8b23-79a5a4202b17"). InnerVolumeSpecName "kube-api-access-dnhkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:36.234445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.234379 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhkj\" (UniqueName: \"kubernetes.io/projected/e328bde8-a577-4594-8b23-79a5a4202b17-kube-api-access-dnhkj\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:36.234445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.234403 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-extensions-socket-volume\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:36.234445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.234416 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbs6\" (UniqueName: \"kubernetes.io/projected/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610-kube-api-access-rrbs6\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:36.241363 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.241329 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:36.249027 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.249004 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.254627 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.254606 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:36.525557 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.525529 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw"] Sep 08 18:46:36.529269 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:36.529239 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7455764_5cd9_4175_9b12_ac5f1ec3f2dc.slice/crio-f30616ab33e3ee05f791f639161601dbc95906e57d6f3586559759842214aaf5 WatchSource:0}: Error finding container f30616ab33e3ee05f791f639161601dbc95906e57d6f3586559759842214aaf5: Status 404 returned error can't find the container with id f30616ab33e3ee05f791f639161601dbc95906e57d6f3586559759842214aaf5 Sep 08 18:46:36.530742 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.530719 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q"] Sep 08 18:46:36.532835 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:36.532814 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5385e112_a64a_4cce_b31a_5a8f307a620e.slice/crio-bd3d4cc1e639fc9406a5fc6218d4f8bb47da196cabe2791b2048f743c62388cc WatchSource:0}: Error finding container bd3d4cc1e639fc9406a5fc6218d4f8bb47da196cabe2791b2048f743c62388cc: Status 404 returned error can't find the container with id bd3d4cc1e639fc9406a5fc6218d4f8bb47da196cabe2791b2048f743c62388cc Sep 08 18:46:36.540705 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.540488 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9"] Sep 08 18:46:36.545792 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:36.545770 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f508a96_5d80_429c_bbc6_c9f1a97560ba.slice/crio-d9c672089d4f197534dd16e7d027c3d525596069a7bfab29d542a39ab7567d4e WatchSource:0}: Error finding container d9c672089d4f197534dd16e7d027c3d525596069a7bfab29d542a39ab7567d4e: Status 404 returned error can't find the container with id d9c672089d4f197534dd16e7d027c3d525596069a7bfab29d542a39ab7567d4e Sep 08 18:46:36.708284 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.708247 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" event={"ID":"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc","Type":"ContainerStarted","Data":"db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4"} Sep 08 18:46:36.708462 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.708301 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" event={"ID":"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc","Type":"ContainerStarted","Data":"f30616ab33e3ee05f791f639161601dbc95906e57d6f3586559759842214aaf5"} Sep 08 18:46:36.708462 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.708328 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:36.709702 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.709675 2557 generic.go:356] "Generic (PLEG): container finished" podID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" containerID="be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4" exitCode=0 Sep 08 18:46:36.709790 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.709702 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zrfwf" Sep 08 18:46:36.709790 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.709760 2557 scope.go:117] "RemoveContainer" containerID="be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4" Sep 08 18:46:36.711503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.711478 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" event={"ID":"5385e112-a64a-4cce-b31a-5a8f307a620e","Type":"ContainerStarted","Data":"256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3"} Sep 08 18:46:36.711597 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.711524 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" event={"ID":"5385e112-a64a-4cce-b31a-5a8f307a620e","Type":"ContainerStarted","Data":"bd3d4cc1e639fc9406a5fc6218d4f8bb47da196cabe2791b2048f743c62388cc"} Sep 08 18:46:36.711597 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.711554 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:36.712943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.712925 2557 generic.go:356] "Generic (PLEG): container finished" podID="e328bde8-a577-4594-8b23-79a5a4202b17" containerID="b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b" exitCode=0 Sep 08 18:46:36.713056 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.712958 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2zbd" Sep 08 18:46:36.715204 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.715183 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" event={"ID":"3f508a96-5d80-429c-bbc6-c9f1a97560ba","Type":"ContainerStarted","Data":"f607de3c285ad784cf16fbd9b76cb36d5a52f1ab8d08d06621f8819cb7ecea57"} Sep 08 18:46:36.715348 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.715328 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:36.715419 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.715372 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" event={"ID":"3f508a96-5d80-429c-bbc6-c9f1a97560ba","Type":"ContainerStarted","Data":"d9c672089d4f197534dd16e7d027c3d525596069a7bfab29d542a39ab7567d4e"} Sep 08 18:46:36.715544 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:36.715531 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" container="manager" Sep 08 18:46:36.720531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.719673 2557 scope.go:117] "RemoveContainer" containerID="be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4" Sep 08 18:46:36.720531 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:36.719992 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4\": container with ID starting with be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4 not found: ID does not exist" containerID="be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4" Sep 08 18:46:36.720531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.720023 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4"} err="failed to get container status \"be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4\": rpc error: code = NotFound desc = could not find container \"be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4\": container with ID starting with be44373e37cdfdbf3d7d1edf1a1bfa07c07f7aa0c44ce540e080beb3e1ba61b4 not found: ID does not exist" Sep 08 18:46:36.720531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.720044 2557 scope.go:117] "RemoveContainer" containerID="b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b" Sep 08 18:46:36.730856 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.730817 2557 scope.go:117] "RemoveContainer" containerID="b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b" Sep 08 18:46:36.731170 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:36.731121 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b\": container with ID starting with b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b not found: ID does not exist" containerID="b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b" Sep 08 18:46:36.731170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.731147 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b"} err="failed to get container status \"b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b\": rpc error: code = NotFound desc = could not find container \"b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b\": container with ID starting with b6699776bfe9b2b97193e5b29f15edca9abc816e4badb990aa1386b06bd1ba9b not found: ID does not exist" Sep 08 18:46:36.770809 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.770766 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" podStartSLOduration=1.7707537979999999 podStartE2EDuration="1.770753798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:36.770032515 +0000 UTC m=+755.279184957" watchObservedRunningTime="2026-09-08 18:46:36.770753798 +0000 UTC m=+755.279906239" Sep 08 18:46:36.807746 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.807578 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" podStartSLOduration=1.807566644 podStartE2EDuration="1.807566644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:36.807554297 +0000 UTC m=+755.316706741" watchObservedRunningTime="2026-09-08 18:46:36.807566644 +0000 UTC m=+755.316719085" Sep 08 18:46:36.862771 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:36.862719 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" podStartSLOduration=1.862704748 podStartE2EDuration="1.862704748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:36.862261478 +0000 UTC m=+755.371413919" watchObservedRunningTime="2026-09-08 18:46:36.862704748 +0000 UTC m=+755.371857190" Sep 08 18:46:37.726193 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:37.726156 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" podUID="1866a50d-1960-4959-912a-301e8cb2805d" containerName="manager" containerID="cri-o://967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e" gracePeriod=2 Sep 08 18:46:37.953410 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:37.953386 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:37.955312 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:37.955286 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-szm8g\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="1866a50d-1960-4959-912a-301e8cb2805d" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:38.024876 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.024821 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610" path="/var/lib/kubelet/pods/d4b8aacf-bf59-4c8c-b7a9-0f4d8a623610/volumes" Sep 08 18:46:38.025118 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.025106 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e328bde8-a577-4594-8b23-79a5a4202b17" path="/var/lib/kubelet/pods/e328bde8-a577-4594-8b23-79a5a4202b17/volumes" Sep 08 18:46:38.047849 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.047827 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcbhv\" (UniqueName: \"kubernetes.io/projected/1866a50d-1960-4959-912a-301e8cb2805d-kube-api-access-dcbhv\") pod \"1866a50d-1960-4959-912a-301e8cb2805d\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " Sep 08 18:46:38.047923 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.047884 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1866a50d-1960-4959-912a-301e8cb2805d-extensions-socket-volume\") pod \"1866a50d-1960-4959-912a-301e8cb2805d\" (UID: \"1866a50d-1960-4959-912a-301e8cb2805d\") " Sep 08 18:46:38.048225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.048202 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1866a50d-1960-4959-912a-301e8cb2805d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1866a50d-1960-4959-912a-301e8cb2805d" (UID: "1866a50d-1960-4959-912a-301e8cb2805d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:38.049709 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.049690 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1866a50d-1960-4959-912a-301e8cb2805d-kube-api-access-dcbhv" (OuterVolumeSpecName: "kube-api-access-dcbhv") pod "1866a50d-1960-4959-912a-301e8cb2805d" (UID: "1866a50d-1960-4959-912a-301e8cb2805d"). InnerVolumeSpecName "kube-api-access-dcbhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:38.149180 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.149159 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1866a50d-1960-4959-912a-301e8cb2805d-extensions-socket-volume\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:38.149180 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.149178 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbhv\" (UniqueName: \"kubernetes.io/projected/1866a50d-1960-4959-912a-301e8cb2805d-kube-api-access-dcbhv\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:38.733075 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.733041 2557 generic.go:356] "Generic (PLEG): container finished" podID="1866a50d-1960-4959-912a-301e8cb2805d" containerID="967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e" exitCode=0 Sep 08 18:46:38.733263 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.733085 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-szm8g" Sep 08 18:46:38.733263 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.733139 2557 scope.go:117] "RemoveContainer" containerID="967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e" Sep 08 18:46:38.741529 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.741514 2557 scope.go:117] "RemoveContainer" containerID="967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e" Sep 08 18:46:38.741797 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:38.741782 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e\": container with ID starting with 967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e not found: ID does not exist" containerID="967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e" Sep 08 18:46:38.741860 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:38.741804 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e"} err="failed to get container status \"967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e\": rpc error: code = NotFound desc = could not find container \"967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e\": container with ID starting with 967b4298c2ed025c4f29f9b578f498cf1aafd349ae488f50ccfc20ddce021d5e not found: ID does not exist" Sep 08 18:46:40.025599 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:40.025560 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1866a50d-1960-4959-912a-301e8cb2805d" path="/var/lib/kubelet/pods/1866a50d-1960-4959-912a-301e8cb2805d/volumes" Sep 08 18:46:47.728113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:47.728083 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:46:47.728529 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:47.728160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fkrd9" Sep 08 18:46:47.728529 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:47.728207 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:47.801035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:47.801004 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q"] Sep 08 18:46:47.801238 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:47.801216 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" podUID="5385e112-a64a-4cce-b31a-5a8f307a620e" containerName="manager" containerID="cri-o://256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3" gracePeriod=10 Sep 08 18:46:48.014846 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.014773 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz"] Sep 08 18:46:48.017497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.017480 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.033149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.033126 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz"] Sep 08 18:46:48.053482 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.053464 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:48.117756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.117732 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6cr\" (UniqueName: \"kubernetes.io/projected/eb0d02b3-5103-44ed-a949-010a77d04f03-kube-api-access-gn6cr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l6qlz\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.117929 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.117779 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0d02b3-5103-44ed-a949-010a77d04f03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l6qlz\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.218481 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.218454 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5385e112-a64a-4cce-b31a-5a8f307a620e-extensions-socket-volume\") pod \"5385e112-a64a-4cce-b31a-5a8f307a620e\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " Sep 08 18:46:48.218596 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.218518 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w472n\" (UniqueName: \"kubernetes.io/projected/5385e112-a64a-4cce-b31a-5a8f307a620e-kube-api-access-w472n\") pod \"5385e112-a64a-4cce-b31a-5a8f307a620e\" (UID: \"5385e112-a64a-4cce-b31a-5a8f307a620e\") " Sep 08 18:46:48.218679 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.218620 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6cr\" (UniqueName: \"kubernetes.io/projected/eb0d02b3-5103-44ed-a949-010a77d04f03-kube-api-access-gn6cr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l6qlz\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.218737 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.218680 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0d02b3-5103-44ed-a949-010a77d04f03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l6qlz\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.218857 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.218805 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5385e112-a64a-4cce-b31a-5a8f307a620e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5385e112-a64a-4cce-b31a-5a8f307a620e" (UID: "5385e112-a64a-4cce-b31a-5a8f307a620e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:48.219048 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.219026 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0d02b3-5103-44ed-a949-010a77d04f03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l6qlz\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.220471 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.220449 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5385e112-a64a-4cce-b31a-5a8f307a620e-kube-api-access-w472n" (OuterVolumeSpecName: "kube-api-access-w472n") pod "5385e112-a64a-4cce-b31a-5a8f307a620e" (UID: "5385e112-a64a-4cce-b31a-5a8f307a620e"). InnerVolumeSpecName "kube-api-access-w472n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:48.227330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.227312 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6cr\" (UniqueName: \"kubernetes.io/projected/eb0d02b3-5103-44ed-a949-010a77d04f03-kube-api-access-gn6cr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l6qlz\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.319633 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.319560 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w472n\" (UniqueName: \"kubernetes.io/projected/5385e112-a64a-4cce-b31a-5a8f307a620e-kube-api-access-w472n\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:48.319633 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.319595 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5385e112-a64a-4cce-b31a-5a8f307a620e-extensions-socket-volume\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:46:48.328560 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.328536 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.611566 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.611539 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz"] Sep 08 18:46:48.613475 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:46:48.613454 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0d02b3_5103_44ed_a949_010a77d04f03.slice/crio-6ce89e90838c7a702a6255279437194cab1b4ceaef0318fbb3d30a6d322ec1cc WatchSource:0}: Error finding container 6ce89e90838c7a702a6255279437194cab1b4ceaef0318fbb3d30a6d322ec1cc: Status 404 returned error can't find the container with id 6ce89e90838c7a702a6255279437194cab1b4ceaef0318fbb3d30a6d322ec1cc Sep 08 18:46:48.790045 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.790014 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" event={"ID":"eb0d02b3-5103-44ed-a949-010a77d04f03","Type":"ContainerStarted","Data":"8a4baf98dd166fe3550d0a1aba31656cbf2b0ef2039ebe5acf6e7708b4f97954"} Sep 08 18:46:48.790448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.790070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" event={"ID":"eb0d02b3-5103-44ed-a949-010a77d04f03","Type":"ContainerStarted","Data":"6ce89e90838c7a702a6255279437194cab1b4ceaef0318fbb3d30a6d322ec1cc"} Sep 08 18:46:48.790448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.790180 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:48.791460 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.791434 2557 generic.go:356] "Generic (PLEG): container finished" podID="5385e112-a64a-4cce-b31a-5a8f307a620e" containerID="256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3" exitCode=0 Sep 08 18:46:48.791532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.791478 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" Sep 08 18:46:48.791532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.791514 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" event={"ID":"5385e112-a64a-4cce-b31a-5a8f307a620e","Type":"ContainerDied","Data":"256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3"} Sep 08 18:46:48.791602 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.791562 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q" event={"ID":"5385e112-a64a-4cce-b31a-5a8f307a620e","Type":"ContainerDied","Data":"bd3d4cc1e639fc9406a5fc6218d4f8bb47da196cabe2791b2048f743c62388cc"} Sep 08 18:46:48.791602 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.791585 2557 scope.go:117] "RemoveContainer" containerID="256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3" Sep 08 18:46:48.799764 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.799749 2557 scope.go:117] "RemoveContainer" containerID="256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3" Sep 08 18:46:48.800096 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:46:48.800076 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3\": container with ID starting with 256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3 not found: ID does not exist" containerID="256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3" Sep 08 18:46:48.800170 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.800103 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3"} err="failed to get container status \"256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3\": rpc error: code = NotFound desc = could not find container \"256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3\": container with ID starting with 256d7fe3319befa5314aae7f712fceb9cdd342d1bbf3be6d0c50ce69ee0d73f3 not found: ID does not exist" Sep 08 18:46:48.816666 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.815140 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" podStartSLOduration=1.815122878 podStartE2EDuration="1.815122878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:48.811745916 +0000 UTC m=+767.320898357" watchObservedRunningTime="2026-09-08 18:46:48.815122878 +0000 UTC m=+767.324275321" Sep 08 18:46:48.828638 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.828616 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q"] Sep 08 18:46:48.834472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:48.834450 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8lh7q"] Sep 08 18:46:50.024154 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:50.024123 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5385e112-a64a-4cce-b31a-5a8f307a620e" path="/var/lib/kubelet/pods/5385e112-a64a-4cce-b31a-5a8f307a620e/volumes" Sep 08 18:46:59.800489 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:59.800462 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 18:46:59.850893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:59.850864 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw"] Sep 08 18:46:59.851158 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:46:59.851131 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" podUID="e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" containerName="manager" containerID="cri-o://db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4" gracePeriod=10 Sep 08 18:47:00.105528 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.105506 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:47:00.210120 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.210094 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqbs8\" (UniqueName: \"kubernetes.io/projected/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-kube-api-access-qqbs8\") pod \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " Sep 08 18:47:00.210219 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.210184 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-extensions-socket-volume\") pod \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\" (UID: \"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc\") " Sep 08 18:47:00.210554 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.210530 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" (UID: "e7455764-5cd9-4175-9b12-ac5f1ec3f2dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:47:00.212048 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.212032 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-kube-api-access-qqbs8" (OuterVolumeSpecName: "kube-api-access-qqbs8") pod "e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" (UID: "e7455764-5cd9-4175-9b12-ac5f1ec3f2dc"). InnerVolumeSpecName "kube-api-access-qqbs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:47:00.311214 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.311191 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbs8\" (UniqueName: \"kubernetes.io/projected/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-kube-api-access-qqbs8\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:00.311293 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.311212 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc-extensions-socket-volume\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:00.852161 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.852126 2557 generic.go:356] "Generic (PLEG): container finished" podID="e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" containerID="db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4" exitCode=0 Sep 08 18:47:00.852516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.852182 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" Sep 08 18:47:00.852516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.852208 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" event={"ID":"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc","Type":"ContainerDied","Data":"db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4"} Sep 08 18:47:00.852516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.852277 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw" event={"ID":"e7455764-5cd9-4175-9b12-ac5f1ec3f2dc","Type":"ContainerDied","Data":"f30616ab33e3ee05f791f639161601dbc95906e57d6f3586559759842214aaf5"} Sep 08 18:47:00.852516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.852298 2557 scope.go:117] "RemoveContainer" containerID="db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4" Sep 08 18:47:00.861762 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.861734 2557 scope.go:117] "RemoveContainer" containerID="db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4" Sep 08 18:47:00.862025 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:47:00.862006 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4\": container with ID starting with db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4 not found: ID does not exist" containerID="db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4" Sep 08 18:47:00.862076 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.862033 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4"} err="failed to get container status \"db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4\": rpc error: code = NotFound desc = could not find container \"db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4\": container with ID starting with db63649a5c4ef3cfcdd7dd222953604d067e770eb29858f568b0640ab83164c4 not found: ID does not exist" Sep 08 18:47:00.875612 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.875591 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw"] Sep 08 18:47:00.879579 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:00.879559 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8fxpw"] Sep 08 18:47:02.024852 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:02.024824 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" path="/var/lib/kubelet/pods/e7455764-5cd9-4175-9b12-ac5f1ec3f2dc/volumes" Sep 08 18:47:04.332363 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332329 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5"] Sep 08 18:47:04.332749 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332694 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" containerName="manager" Sep 08 18:47:04.332749 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332707 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" containerName="manager" Sep 08 18:47:04.332749 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332727 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5385e112-a64a-4cce-b31a-5a8f307a620e" containerName="manager" Sep 08 18:47:04.332749 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332732 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5385e112-a64a-4cce-b31a-5a8f307a620e" containerName="manager" Sep 08 18:47:04.332881 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332814 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5385e112-a64a-4cce-b31a-5a8f307a620e" containerName="manager" Sep 08 18:47:04.332881 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.332825 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7455764-5cd9-4175-9b12-ac5f1ec3f2dc" containerName="manager" Sep 08 18:47:04.337731 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.337700 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.340124 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.340098 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-df2pm\"" Sep 08 18:47:04.340247 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.340105 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 18:47:04.345200 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.345172 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5"] Sep 08 18:47:04.440635 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440841 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440841 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw87v\" (UniqueName: \"kubernetes.io/projected/9dc57368-c1fe-40d7-bf89-4b4145a7497e-kube-api-access-rw87v\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440841 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440807 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440841 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440833 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440985 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440866 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440985 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440985 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.440985 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.440974 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.541453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541430 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.541558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541456 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw87v\" (UniqueName: \"kubernetes.io/projected/9dc57368-c1fe-40d7-bf89-4b4145a7497e-kube-api-access-rw87v\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.541558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.541558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541493 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.541558 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.541812 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542002 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541979 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541992 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.541995 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542224 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.542121 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542224 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.542000 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542224 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.542168 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.542405 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.542755 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.542736 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.543827 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.543809 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.544126 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.544107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.548995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.548972 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc57368-c1fe-40d7-bf89-4b4145a7497e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.549066 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.549040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw87v\" (UniqueName: \"kubernetes.io/projected/9dc57368-c1fe-40d7-bf89-4b4145a7497e-kube-api-access-rw87v\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wf9s5\" (UID: \"9dc57368-c1fe-40d7-bf89-4b4145a7497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.650119 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.650069 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:04.920531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:04.920439 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5"] Sep 08 18:47:04.922698 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:47:04.922675 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc57368_c1fe_40d7_bf89_4b4145a7497e.slice/crio-da6dcc49903fd3a1220051c52e5e9b286dc456debcb2ce16649f595b9b5f5fc2 WatchSource:0}: Error finding container da6dcc49903fd3a1220051c52e5e9b286dc456debcb2ce16649f595b9b5f5fc2: Status 404 returned error can't find the container with id da6dcc49903fd3a1220051c52e5e9b286dc456debcb2ce16649f595b9b5f5fc2 Sep 08 18:47:05.877360 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:05.877323 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" event={"ID":"9dc57368-c1fe-40d7-bf89-4b4145a7497e","Type":"ContainerStarted","Data":"da6dcc49903fd3a1220051c52e5e9b286dc456debcb2ce16649f595b9b5f5fc2"} Sep 08 18:47:08.781192 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:08.781152 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:47:08.781481 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:08.781232 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:47:08.781481 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:08.781264 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:47:08.895584 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:08.895549 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" event={"ID":"9dc57368-c1fe-40d7-bf89-4b4145a7497e","Type":"ContainerStarted","Data":"2f8b879fa1d9185067c4757d69ce68e9fea2870a5fdd64df6be9bca432a6f0cf"} Sep 08 18:47:08.913607 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:08.913564 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podStartSLOduration=1.057502298 podStartE2EDuration="4.913548944s" totalImagesPullingTime="3.856046646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:08.911911274 +0000 UTC m=+787.421063715" watchObservedRunningTime="2026-09-08 18:47:08.913548944 +0000 UTC m=+787.422701385" Sep 08 18:47:09.651189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:09.651159 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:09.655413 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:09.655390 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:09.899573 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:09.899550 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:09.900417 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:09.900398 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:47:19.866582 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:19.866544 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:19.870235 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:19.870211 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:19.872438 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:19.872419 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 18:47:19.872534 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:19.872461 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mqx9b\"" Sep 08 18:47:19.876514 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:19.876494 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:19.973695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:19.973663 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:20.053828 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.053797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhd4\" (UniqueName: \"kubernetes.io/projected/2b2edf73-7177-4e7c-8452-5eec84ed9a15-kube-api-access-8lhd4\") pod \"limitador-limitador-8588749977-qtpqp\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.053828 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.053834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2b2edf73-7177-4e7c-8452-5eec84ed9a15-config-file\") pod \"limitador-limitador-8588749977-qtpqp\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.154843 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.154760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhd4\" (UniqueName: \"kubernetes.io/projected/2b2edf73-7177-4e7c-8452-5eec84ed9a15-kube-api-access-8lhd4\") pod \"limitador-limitador-8588749977-qtpqp\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.154991 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.154913 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2b2edf73-7177-4e7c-8452-5eec84ed9a15-config-file\") pod \"limitador-limitador-8588749977-qtpqp\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.155530 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.155510 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2b2edf73-7177-4e7c-8452-5eec84ed9a15-config-file\") pod \"limitador-limitador-8588749977-qtpqp\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.162902 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.162877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhd4\" (UniqueName: \"kubernetes.io/projected/2b2edf73-7177-4e7c-8452-5eec84ed9a15-kube-api-access-8lhd4\") pod \"limitador-limitador-8588749977-qtpqp\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.180680 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.180654 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:20.454904 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.454880 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:20.457127 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:47:20.457104 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2edf73_7177_4e7c_8452_5eec84ed9a15.slice/crio-7254a6083eac56c3591e801d918e68d8a0d8a9bf5fb134ecee0082ad75fb7db2 WatchSource:0}: Error finding container 7254a6083eac56c3591e801d918e68d8a0d8a9bf5fb134ecee0082ad75fb7db2: Status 404 returned error can't find the container with id 7254a6083eac56c3591e801d918e68d8a0d8a9bf5fb134ecee0082ad75fb7db2 Sep 08 18:47:20.656369 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.656344 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-94xk4"] Sep 08 18:47:20.660994 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.660974 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:20.662968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.662950 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9vfpz\"" Sep 08 18:47:20.665701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.665680 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-94xk4"] Sep 08 18:47:20.761158 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.761130 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgp6\" (UniqueName: \"kubernetes.io/projected/400e056b-838d-4cf7-8376-5a38b05f3f04-kube-api-access-chgp6\") pod \"authorino-7bcfd8f9cd-94xk4\" (UID: \"400e056b-838d-4cf7-8376-5a38b05f3f04\") " pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:20.772584 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.772560 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-7sbqw"] Sep 08 18:47:20.775896 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.775881 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:20.782323 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.782301 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7sbqw"] Sep 08 18:47:20.861591 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.861556 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqqj\" (UniqueName: \"kubernetes.io/projected/b5130ac8-5c7c-478e-bd86-8654322b61c5-kube-api-access-pzqqj\") pod \"authorino-59c974bd48-7sbqw\" (UID: \"b5130ac8-5c7c-478e-bd86-8654322b61c5\") " pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:20.861725 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.861597 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgp6\" (UniqueName: \"kubernetes.io/projected/400e056b-838d-4cf7-8376-5a38b05f3f04-kube-api-access-chgp6\") pod \"authorino-7bcfd8f9cd-94xk4\" (UID: \"400e056b-838d-4cf7-8376-5a38b05f3f04\") " pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:20.869409 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.869379 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgp6\" (UniqueName: \"kubernetes.io/projected/400e056b-838d-4cf7-8376-5a38b05f3f04-kube-api-access-chgp6\") pod \"authorino-7bcfd8f9cd-94xk4\" (UID: \"400e056b-838d-4cf7-8376-5a38b05f3f04\") " pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:20.951093 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.951067 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" event={"ID":"2b2edf73-7177-4e7c-8452-5eec84ed9a15","Type":"ContainerStarted","Data":"7254a6083eac56c3591e801d918e68d8a0d8a9bf5fb134ecee0082ad75fb7db2"} Sep 08 18:47:20.961932 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.961904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqqj\" (UniqueName: \"kubernetes.io/projected/b5130ac8-5c7c-478e-bd86-8654322b61c5-kube-api-access-pzqqj\") pod \"authorino-59c974bd48-7sbqw\" (UID: \"b5130ac8-5c7c-478e-bd86-8654322b61c5\") " pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:20.969254 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.969234 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqqj\" (UniqueName: \"kubernetes.io/projected/b5130ac8-5c7c-478e-bd86-8654322b61c5-kube-api-access-pzqqj\") pod \"authorino-59c974bd48-7sbqw\" (UID: \"b5130ac8-5c7c-478e-bd86-8654322b61c5\") " pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:20.971039 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:20.971023 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:21.086255 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.086228 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:21.237030 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.237001 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-94xk4"] Sep 08 18:47:21.239004 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:47:21.238980 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod400e056b_838d_4cf7_8376_5a38b05f3f04.slice/crio-fa62d76c98ebca28fbd981e1855703e8af44799481da16d7f695f441b8e6c051 WatchSource:0}: Error finding container fa62d76c98ebca28fbd981e1855703e8af44799481da16d7f695f441b8e6c051: Status 404 returned error can't find the container with id fa62d76c98ebca28fbd981e1855703e8af44799481da16d7f695f441b8e6c051 Sep 08 18:47:21.358428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.358402 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7sbqw"] Sep 08 18:47:21.435239 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:47:21.435205 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5130ac8_5c7c_478e_bd86_8654322b61c5.slice/crio-40cecab6c16e8bbec806eae8e06ed3275ba7a90a40485c385e657e7871181e58 WatchSource:0}: Error finding container 40cecab6c16e8bbec806eae8e06ed3275ba7a90a40485c385e657e7871181e58: Status 404 returned error can't find the container with id 40cecab6c16e8bbec806eae8e06ed3275ba7a90a40485c385e657e7871181e58 Sep 08 18:47:21.957687 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.957640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" event={"ID":"2b2edf73-7177-4e7c-8452-5eec84ed9a15","Type":"ContainerStarted","Data":"a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f"} Sep 08 18:47:21.958071 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.957743 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:21.958932 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.958910 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7sbqw" event={"ID":"b5130ac8-5c7c-478e-bd86-8654322b61c5","Type":"ContainerStarted","Data":"40cecab6c16e8bbec806eae8e06ed3275ba7a90a40485c385e657e7871181e58"} Sep 08 18:47:21.960100 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.960081 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" event={"ID":"400e056b-838d-4cf7-8376-5a38b05f3f04","Type":"ContainerStarted","Data":"fa62d76c98ebca28fbd981e1855703e8af44799481da16d7f695f441b8e6c051"} Sep 08 18:47:21.974495 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:21.974452 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" podStartSLOduration=1.9433294509999999 podStartE2EDuration="2.974439281s" totalImagesPullingTime="1.03110983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:21.97204485 +0000 UTC m=+800.481197303" watchObservedRunningTime="2026-09-08 18:47:21.974439281 +0000 UTC m=+800.483591722" Sep 08 18:47:23.973385 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:23.973347 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7sbqw" event={"ID":"b5130ac8-5c7c-478e-bd86-8654322b61c5","Type":"ContainerStarted","Data":"edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a"} Sep 08 18:47:23.974809 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:23.974783 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" event={"ID":"400e056b-838d-4cf7-8376-5a38b05f3f04","Type":"ContainerStarted","Data":"cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7"} Sep 08 18:47:23.986207 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:23.986161 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-7sbqw" podStartSLOduration=2.316899327 podStartE2EDuration="3.986148833s" totalImagesPullingTime="1.669249506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:23.985497755 +0000 UTC m=+802.494650197" watchObservedRunningTime="2026-09-08 18:47:23.986148833 +0000 UTC m=+802.495301273" Sep 08 18:47:24.012613 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:24.012572 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" podStartSLOduration=2.151688892 podStartE2EDuration="4.012557326s" totalImagesPullingTime="1.860868434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:24.010433648 +0000 UTC m=+802.519586088" watchObservedRunningTime="2026-09-08 18:47:24.012557326 +0000 UTC m=+802.521709765" Sep 08 18:47:24.076535 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:24.076506 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-94xk4"] Sep 08 18:47:25.985297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:25.985243 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" podUID="400e056b-838d-4cf7-8376-5a38b05f3f04" containerName="authorino" containerID="cri-o://cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7" gracePeriod=30 Sep 08 18:47:26.239743 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.239694 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:26.300894 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.300868 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chgp6\" (UniqueName: \"kubernetes.io/projected/400e056b-838d-4cf7-8376-5a38b05f3f04-kube-api-access-chgp6\") pod \"400e056b-838d-4cf7-8376-5a38b05f3f04\" (UID: \"400e056b-838d-4cf7-8376-5a38b05f3f04\") " Sep 08 18:47:26.302765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.302738 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400e056b-838d-4cf7-8376-5a38b05f3f04-kube-api-access-chgp6" (OuterVolumeSpecName: "kube-api-access-chgp6") pod "400e056b-838d-4cf7-8376-5a38b05f3f04" (UID: "400e056b-838d-4cf7-8376-5a38b05f3f04"). InnerVolumeSpecName "kube-api-access-chgp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:47:26.401528 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.401503 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chgp6\" (UniqueName: \"kubernetes.io/projected/400e056b-838d-4cf7-8376-5a38b05f3f04-kube-api-access-chgp6\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:26.991403 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.991368 2557 generic.go:356] "Generic (PLEG): container finished" podID="400e056b-838d-4cf7-8376-5a38b05f3f04" containerID="cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7" exitCode=0 Sep 08 18:47:26.991853 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.991409 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" Sep 08 18:47:26.991853 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.991448 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" event={"ID":"400e056b-838d-4cf7-8376-5a38b05f3f04","Type":"ContainerDied","Data":"cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7"} Sep 08 18:47:26.991853 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.991486 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-94xk4" event={"ID":"400e056b-838d-4cf7-8376-5a38b05f3f04","Type":"ContainerDied","Data":"fa62d76c98ebca28fbd981e1855703e8af44799481da16d7f695f441b8e6c051"} Sep 08 18:47:26.991853 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:26.991503 2557 scope.go:117] "RemoveContainer" containerID="cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7" Sep 08 18:47:27.002189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:27.002172 2557 scope.go:117] "RemoveContainer" containerID="cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7" Sep 08 18:47:27.002432 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:47:27.002415 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7\": container with ID starting with cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7 not found: ID does not exist" containerID="cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7" Sep 08 18:47:27.002493 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:27.002444 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7"} err="failed to get container status \"cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7\": rpc error: code = NotFound desc = could not find container \"cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7\": container with ID starting with cf007bddedaead79585a6d61d7941cf1d8a33af4130faa52ab5f975d13d9d1b7 not found: ID does not exist" Sep 08 18:47:27.012222 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:27.012201 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-94xk4"] Sep 08 18:47:27.015702 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:27.015681 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-94xk4"] Sep 08 18:47:28.024577 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:28.024547 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400e056b-838d-4cf7-8376-5a38b05f3f04" path="/var/lib/kubelet/pods/400e056b-838d-4cf7-8376-5a38b05f3f04/volumes" Sep 08 18:47:32.968487 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:32.968451 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:35.222851 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.222817 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:35.223271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.223019 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" podUID="2b2edf73-7177-4e7c-8452-5eec84ed9a15" containerName="limitador" containerID="cri-o://a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f" gracePeriod=30 Sep 08 18:47:35.770538 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.770512 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:35.867216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.867147 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2b2edf73-7177-4e7c-8452-5eec84ed9a15-config-file\") pod \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " Sep 08 18:47:35.867216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.867183 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lhd4\" (UniqueName: \"kubernetes.io/projected/2b2edf73-7177-4e7c-8452-5eec84ed9a15-kube-api-access-8lhd4\") pod \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\" (UID: \"2b2edf73-7177-4e7c-8452-5eec84ed9a15\") " Sep 08 18:47:35.867492 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.867467 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2edf73-7177-4e7c-8452-5eec84ed9a15-config-file" (OuterVolumeSpecName: "config-file") pod "2b2edf73-7177-4e7c-8452-5eec84ed9a15" (UID: "2b2edf73-7177-4e7c-8452-5eec84ed9a15"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:47:35.869194 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.869166 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2edf73-7177-4e7c-8452-5eec84ed9a15-kube-api-access-8lhd4" (OuterVolumeSpecName: "kube-api-access-8lhd4") pod "2b2edf73-7177-4e7c-8452-5eec84ed9a15" (UID: "2b2edf73-7177-4e7c-8452-5eec84ed9a15"). InnerVolumeSpecName "kube-api-access-8lhd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:47:35.967751 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.967724 2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2b2edf73-7177-4e7c-8452-5eec84ed9a15-config-file\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:35.967751 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:35.967747 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhd4\" (UniqueName: \"kubernetes.io/projected/2b2edf73-7177-4e7c-8452-5eec84ed9a15-kube-api-access-8lhd4\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:36.036029 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.036003 2557 generic.go:356] "Generic (PLEG): container finished" podID="2b2edf73-7177-4e7c-8452-5eec84ed9a15" containerID="a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f" exitCode=0 Sep 08 18:47:36.036119 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.036041 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" Sep 08 18:47:36.036119 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.036076 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" event={"ID":"2b2edf73-7177-4e7c-8452-5eec84ed9a15","Type":"ContainerDied","Data":"a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f"} Sep 08 18:47:36.036194 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.036123 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qtpqp" event={"ID":"2b2edf73-7177-4e7c-8452-5eec84ed9a15","Type":"ContainerDied","Data":"7254a6083eac56c3591e801d918e68d8a0d8a9bf5fb134ecee0082ad75fb7db2"} Sep 08 18:47:36.036194 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.036145 2557 scope.go:117] "RemoveContainer" containerID="a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f" Sep 08 18:47:36.044475 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.044454 2557 scope.go:117] "RemoveContainer" containerID="a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f" Sep 08 18:47:36.044821 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:47:36.044801 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f\": container with ID starting with a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f not found: ID does not exist" containerID="a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f" Sep 08 18:47:36.044896 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.044832 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f"} err="failed to get container status \"a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f\": rpc error: code = NotFound desc = could not find container \"a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f\": container with ID starting with a7c13f593f8158e4be09e5acdf8aa852aee0339340edc3c90ec32cd3ea343b3f not found: ID does not exist" Sep 08 18:47:36.050564 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.050544 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:36.055157 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.055139 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qtpqp"] Sep 08 18:47:36.894297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894265 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t"] Sep 08 18:47:36.894811 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894793 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400e056b-838d-4cf7-8376-5a38b05f3f04" containerName="authorino" Sep 08 18:47:36.894892 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894814 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="400e056b-838d-4cf7-8376-5a38b05f3f04" containerName="authorino" Sep 08 18:47:36.894892 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894839 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2edf73-7177-4e7c-8452-5eec84ed9a15" containerName="limitador" Sep 08 18:47:36.894892 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894847 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2edf73-7177-4e7c-8452-5eec84ed9a15" containerName="limitador" Sep 08 18:47:36.895035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894948 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="400e056b-838d-4cf7-8376-5a38b05f3f04" containerName="authorino" Sep 08 18:47:36.895035 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.894963 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2edf73-7177-4e7c-8452-5eec84ed9a15" containerName="limitador" Sep 08 18:47:36.899579 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.899560 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:36.902081 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.902055 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 08 18:47:36.902081 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.902076 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 18:47:36.902241 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.902107 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:47:36.902841 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.902823 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-f8qnh\"" Sep 08 18:47:36.915158 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.915140 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t"] Sep 08 18:47:36.973898 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.973873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b3776b0-0796-4d57-80ec-20fbdd65fd29-data\") pod \"postgres-69fb4594cf-dpx9t\" (UID: \"8b3776b0-0796-4d57-80ec-20fbdd65fd29\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:36.973981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:36.973913 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqd8\" (UniqueName: \"kubernetes.io/projected/8b3776b0-0796-4d57-80ec-20fbdd65fd29-kube-api-access-fpqd8\") pod \"postgres-69fb4594cf-dpx9t\" (UID: \"8b3776b0-0796-4d57-80ec-20fbdd65fd29\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:37.074634 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:37.074605 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b3776b0-0796-4d57-80ec-20fbdd65fd29-data\") pod \"postgres-69fb4594cf-dpx9t\" (UID: \"8b3776b0-0796-4d57-80ec-20fbdd65fd29\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:37.074773 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:37.074672 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqd8\" (UniqueName: \"kubernetes.io/projected/8b3776b0-0796-4d57-80ec-20fbdd65fd29-kube-api-access-fpqd8\") pod \"postgres-69fb4594cf-dpx9t\" (UID: \"8b3776b0-0796-4d57-80ec-20fbdd65fd29\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:37.074997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:37.074977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b3776b0-0796-4d57-80ec-20fbdd65fd29-data\") pod \"postgres-69fb4594cf-dpx9t\" (UID: \"8b3776b0-0796-4d57-80ec-20fbdd65fd29\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:37.083396 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:37.083366 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqd8\" (UniqueName: \"kubernetes.io/projected/8b3776b0-0796-4d57-80ec-20fbdd65fd29-kube-api-access-fpqd8\") pod \"postgres-69fb4594cf-dpx9t\" (UID: \"8b3776b0-0796-4d57-80ec-20fbdd65fd29\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:37.210727 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:37.210672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:37.477709 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:37.477633 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t"] Sep 08 18:47:37.479876 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:47:37.479840 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3776b0_0796_4d57_80ec_20fbdd65fd29.slice/crio-8873dedb2b36545df326e05099c5446f8ef7e72389085fd418132563e43af588 WatchSource:0}: Error finding container 8873dedb2b36545df326e05099c5446f8ef7e72389085fd418132563e43af588: Status 404 returned error can't find the container with id 8873dedb2b36545df326e05099c5446f8ef7e72389085fd418132563e43af588 Sep 08 18:47:38.024576 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:38.024544 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2edf73-7177-4e7c-8452-5eec84ed9a15" path="/var/lib/kubelet/pods/2b2edf73-7177-4e7c-8452-5eec84ed9a15/volumes" Sep 08 18:47:38.046462 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:38.046432 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" event={"ID":"8b3776b0-0796-4d57-80ec-20fbdd65fd29","Type":"ContainerStarted","Data":"8873dedb2b36545df326e05099c5446f8ef7e72389085fd418132563e43af588"} Sep 08 18:47:43.077633 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:43.077594 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" event={"ID":"8b3776b0-0796-4d57-80ec-20fbdd65fd29","Type":"ContainerStarted","Data":"a285e1ddc793c50e944f3a9d194fe8153e2fd29ce59d166129f37d5d08687eb5"} Sep 08 18:47:43.078128 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:43.077678 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:43.093127 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:43.093072 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" podStartSLOduration=2.23586922 podStartE2EDuration="7.093057622s" totalImagesPullingTime="4.857188402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:43.091683444 +0000 UTC m=+821.600835884" watchObservedRunningTime="2026-09-08 18:47:43.093057622 +0000 UTC m=+821.602210064" Sep 08 18:47:49.112445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.112417 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dpx9t" Sep 08 18:47:49.687915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.687878 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-lqjst"] Sep 08 18:47:49.690042 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.690025 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:49.711347 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.711320 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-lqjst"] Sep 08 18:47:49.768450 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.768420 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kh44\" (UniqueName: \"kubernetes.io/projected/cff2275f-fc20-4069-93c1-ca79230d3c70-kube-api-access-4kh44\") pod \"authorino-557f876f69-lqjst\" (UID: \"cff2275f-fc20-4069-93c1-ca79230d3c70\") " pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:49.813110 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.813087 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lqjst"] Sep 08 18:47:49.813317 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:47:49.813297 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4kh44], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-lqjst" podUID="cff2275f-fc20-4069-93c1-ca79230d3c70" Sep 08 18:47:49.838199 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.838177 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-cwzbd"] Sep 08 18:47:49.840229 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.840213 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:49.842394 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.842369 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 18:47:49.847504 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.847486 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-cwzbd"] Sep 08 18:47:49.869186 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.869163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kh44\" (UniqueName: \"kubernetes.io/projected/cff2275f-fc20-4069-93c1-ca79230d3c70-kube-api-access-4kh44\") pod \"authorino-557f876f69-lqjst\" (UID: \"cff2275f-fc20-4069-93c1-ca79230d3c70\") " pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:49.869276 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.869207 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc8cf771-ccff-418e-b4b7-4be47dc6293c-tls-cert\") pod \"authorino-6d4dfc649f-cwzbd\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:49.869276 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.869241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhd7\" (UniqueName: \"kubernetes.io/projected/dc8cf771-ccff-418e-b4b7-4be47dc6293c-kube-api-access-klhd7\") pod \"authorino-6d4dfc649f-cwzbd\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:49.876453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.876429 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kh44\" (UniqueName: \"kubernetes.io/projected/cff2275f-fc20-4069-93c1-ca79230d3c70-kube-api-access-4kh44\") pod \"authorino-557f876f69-lqjst\" (UID: \"cff2275f-fc20-4069-93c1-ca79230d3c70\") " pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:49.910100 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.910077 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-cwzbd"] Sep 08 18:47:49.910285 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:47:49.910268 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-klhd7 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" podUID="dc8cf771-ccff-418e-b4b7-4be47dc6293c" Sep 08 18:47:49.942412 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.942348 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-598d6898d-8pgtt"] Sep 08 18:47:49.944425 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.944407 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:49.954348 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.954328 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-598d6898d-8pgtt"] Sep 08 18:47:49.970569 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.970542 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc8cf771-ccff-418e-b4b7-4be47dc6293c-tls-cert\") pod \"authorino-6d4dfc649f-cwzbd\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:49.970692 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.970574 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx9p\" (UniqueName: \"kubernetes.io/projected/fec268dc-448a-4318-9676-151598e2cfb9-kube-api-access-8nx9p\") pod \"authorino-598d6898d-8pgtt\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:49.970692 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.970604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhd7\" (UniqueName: \"kubernetes.io/projected/dc8cf771-ccff-418e-b4b7-4be47dc6293c-kube-api-access-klhd7\") pod \"authorino-6d4dfc649f-cwzbd\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:49.970692 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.970636 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fec268dc-448a-4318-9676-151598e2cfb9-tls-cert\") pod \"authorino-598d6898d-8pgtt\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:49.972862 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.972842 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc8cf771-ccff-418e-b4b7-4be47dc6293c-tls-cert\") pod \"authorino-6d4dfc649f-cwzbd\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:49.977357 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:49.977336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhd7\" (UniqueName: \"kubernetes.io/projected/dc8cf771-ccff-418e-b4b7-4be47dc6293c-kube-api-access-klhd7\") pod \"authorino-6d4dfc649f-cwzbd\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:50.071401 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.071378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fec268dc-448a-4318-9676-151598e2cfb9-tls-cert\") pod \"authorino-598d6898d-8pgtt\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:50.071489 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.071442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx9p\" (UniqueName: \"kubernetes.io/projected/fec268dc-448a-4318-9676-151598e2cfb9-kube-api-access-8nx9p\") pod \"authorino-598d6898d-8pgtt\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:50.073540 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.073520 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fec268dc-448a-4318-9676-151598e2cfb9-tls-cert\") pod \"authorino-598d6898d-8pgtt\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:50.078580 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.078559 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx9p\" (UniqueName: \"kubernetes.io/projected/fec268dc-448a-4318-9676-151598e2cfb9-kube-api-access-8nx9p\") pod \"authorino-598d6898d-8pgtt\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:50.117377 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.117349 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:50.117691 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.117352 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:50.123018 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.123000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:50.125777 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.125761 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:50.171889 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.171871 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klhd7\" (UniqueName: \"kubernetes.io/projected/dc8cf771-ccff-418e-b4b7-4be47dc6293c-kube-api-access-klhd7\") pod \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " Sep 08 18:47:50.172009 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.171909 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc8cf771-ccff-418e-b4b7-4be47dc6293c-tls-cert\") pod \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\" (UID: \"dc8cf771-ccff-418e-b4b7-4be47dc6293c\") " Sep 08 18:47:50.172009 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.171944 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kh44\" (UniqueName: \"kubernetes.io/projected/cff2275f-fc20-4069-93c1-ca79230d3c70-kube-api-access-4kh44\") pod \"cff2275f-fc20-4069-93c1-ca79230d3c70\" (UID: \"cff2275f-fc20-4069-93c1-ca79230d3c70\") " Sep 08 18:47:50.173748 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.173724 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8cf771-ccff-418e-b4b7-4be47dc6293c-kube-api-access-klhd7" (OuterVolumeSpecName: "kube-api-access-klhd7") pod "dc8cf771-ccff-418e-b4b7-4be47dc6293c" (UID: "dc8cf771-ccff-418e-b4b7-4be47dc6293c"). InnerVolumeSpecName "kube-api-access-klhd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:47:50.173875 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.173854 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff2275f-fc20-4069-93c1-ca79230d3c70-kube-api-access-4kh44" (OuterVolumeSpecName: "kube-api-access-4kh44") pod "cff2275f-fc20-4069-93c1-ca79230d3c70" (UID: "cff2275f-fc20-4069-93c1-ca79230d3c70"). InnerVolumeSpecName "kube-api-access-4kh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:47:50.173875 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.173865 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8cf771-ccff-418e-b4b7-4be47dc6293c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dc8cf771-ccff-418e-b4b7-4be47dc6293c" (UID: "dc8cf771-ccff-418e-b4b7-4be47dc6293c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:47:50.254190 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.254166 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:47:50.272453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.272432 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kh44\" (UniqueName: \"kubernetes.io/projected/cff2275f-fc20-4069-93c1-ca79230d3c70-kube-api-access-4kh44\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:50.272543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.272455 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhd7\" (UniqueName: \"kubernetes.io/projected/dc8cf771-ccff-418e-b4b7-4be47dc6293c-kube-api-access-klhd7\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:50.272543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.272465 2557 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc8cf771-ccff-418e-b4b7-4be47dc6293c-tls-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:50.521983 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:50.521896 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-598d6898d-8pgtt"] Sep 08 18:47:50.528112 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:47:50.528089 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfec268dc_448a_4318_9676_151598e2cfb9.slice/crio-198ab6de044d83ec03d328899886fce7e55ab46c48bfa6d5a96ecac4ca07ed09 WatchSource:0}: Error finding container 198ab6de044d83ec03d328899886fce7e55ab46c48bfa6d5a96ecac4ca07ed09: Status 404 returned error can't find the container with id 198ab6de044d83ec03d328899886fce7e55ab46c48bfa6d5a96ecac4ca07ed09 Sep 08 18:47:51.123219 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.123183 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-598d6898d-8pgtt" event={"ID":"fec268dc-448a-4318-9676-151598e2cfb9","Type":"ContainerStarted","Data":"9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091"} Sep 08 18:47:51.123219 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.123221 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lqjst" Sep 08 18:47:51.123219 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.123232 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-598d6898d-8pgtt" event={"ID":"fec268dc-448a-4318-9676-151598e2cfb9","Type":"ContainerStarted","Data":"198ab6de044d83ec03d328899886fce7e55ab46c48bfa6d5a96ecac4ca07ed09"} Sep 08 18:47:51.123673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.123391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-cwzbd" Sep 08 18:47:51.141573 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.141498 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-598d6898d-8pgtt" podStartSLOduration=1.726046147 podStartE2EDuration="2.141480589s" totalImagesPullingTime="415.434442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:51.140028127 +0000 UTC m=+829.649180569" watchObservedRunningTime="2026-09-08 18:47:51.141480589 +0000 UTC m=+829.650633030" Sep 08 18:47:51.163361 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.163338 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lqjst"] Sep 08 18:47:51.170002 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.169978 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-lqjst"] Sep 08 18:47:51.172483 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.172460 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7sbqw"] Sep 08 18:47:51.172717 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.172668 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-7sbqw" podUID="b5130ac8-5c7c-478e-bd86-8654322b61c5" containerName="authorino" containerID="cri-o://edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a" gracePeriod=30 Sep 08 18:47:51.192531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.192507 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-cwzbd"] Sep 08 18:47:51.196139 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.196112 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-cwzbd"] Sep 08 18:47:51.399347 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.399326 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:51.482020 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.481989 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzqqj\" (UniqueName: \"kubernetes.io/projected/b5130ac8-5c7c-478e-bd86-8654322b61c5-kube-api-access-pzqqj\") pod \"b5130ac8-5c7c-478e-bd86-8654322b61c5\" (UID: \"b5130ac8-5c7c-478e-bd86-8654322b61c5\") " Sep 08 18:47:51.483856 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.483836 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5130ac8-5c7c-478e-bd86-8654322b61c5-kube-api-access-pzqqj" (OuterVolumeSpecName: "kube-api-access-pzqqj") pod "b5130ac8-5c7c-478e-bd86-8654322b61c5" (UID: "b5130ac8-5c7c-478e-bd86-8654322b61c5"). InnerVolumeSpecName "kube-api-access-pzqqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:47:51.582944 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:51.582918 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqqj\" (UniqueName: \"kubernetes.io/projected/b5130ac8-5c7c-478e-bd86-8654322b61c5-kube-api-access-pzqqj\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:47:52.024687 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.024662 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff2275f-fc20-4069-93c1-ca79230d3c70" path="/var/lib/kubelet/pods/cff2275f-fc20-4069-93c1-ca79230d3c70/volumes" Sep 08 18:47:52.024907 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.024896 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc8cf771-ccff-418e-b4b7-4be47dc6293c" path="/var/lib/kubelet/pods/dc8cf771-ccff-418e-b4b7-4be47dc6293c/volumes" Sep 08 18:47:52.128586 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.128561 2557 generic.go:356] "Generic (PLEG): container finished" podID="b5130ac8-5c7c-478e-bd86-8654322b61c5" containerID="edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a" exitCode=0 Sep 08 18:47:52.128903 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.128595 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7sbqw" Sep 08 18:47:52.128903 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.128622 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7sbqw" event={"ID":"b5130ac8-5c7c-478e-bd86-8654322b61c5","Type":"ContainerDied","Data":"edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a"} Sep 08 18:47:52.128903 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.128668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7sbqw" event={"ID":"b5130ac8-5c7c-478e-bd86-8654322b61c5","Type":"ContainerDied","Data":"40cecab6c16e8bbec806eae8e06ed3275ba7a90a40485c385e657e7871181e58"} Sep 08 18:47:52.128903 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.128685 2557 scope.go:117] "RemoveContainer" containerID="edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a" Sep 08 18:47:52.136750 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.136736 2557 scope.go:117] "RemoveContainer" containerID="edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a" Sep 08 18:47:52.136980 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:47:52.136959 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a\": container with ID starting with edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a not found: ID does not exist" containerID="edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a" Sep 08 18:47:52.137071 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.136991 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a"} err="failed to get container status \"edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a\": rpc error: code = NotFound desc = could not find container \"edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a\": container with ID starting with edae178af1f120b34dee0cd5043355c2a4f9b316a6b2821889ec46a9fc9dcb7a not found: ID does not exist" Sep 08 18:47:52.144830 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.144808 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7sbqw"] Sep 08 18:47:52.148708 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:52.148689 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7sbqw"] Sep 08 18:47:54.024938 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:47:54.024910 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5130ac8-5c7c-478e-bd86-8654322b61c5" path="/var/lib/kubelet/pods/b5130ac8-5c7c-478e-bd86-8654322b61c5/volumes" Sep 08 18:48:32.404900 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.404865 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf"] Sep 08 18:48:32.405451 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.405384 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5130ac8-5c7c-478e-bd86-8654322b61c5" containerName="authorino" Sep 08 18:48:32.405451 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.405403 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5130ac8-5c7c-478e-bd86-8654322b61c5" containerName="authorino" Sep 08 18:48:32.405569 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.405512 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5130ac8-5c7c-478e-bd86-8654322b61c5" containerName="authorino" Sep 08 18:48:32.408453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.408432 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.411476 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.411452 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 18:48:32.411581 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.411475 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 18:48:32.411581 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.411511 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\"" Sep 08 18:48:32.418366 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.418345 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf"] Sep 08 18:48:32.476590 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.476566 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.476735 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.476603 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvsz\" (UniqueName: \"kubernetes.io/projected/16722a0a-ba23-46c7-889f-f940d3083445-kube-api-access-hbvsz\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.476735 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.476628 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-metrics-certs\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.476735 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.476674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16722a0a-ba23-46c7-889f-f940d3083445-openshift-service-ca\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.577529 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.577503 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.577636 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.577548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvsz\" (UniqueName: \"kubernetes.io/projected/16722a0a-ba23-46c7-889f-f940d3083445-kube-api-access-hbvsz\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.577636 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.577569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-metrics-certs\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.577636 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.577584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16722a0a-ba23-46c7-889f-f940d3083445-openshift-service-ca\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.577836 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:48:32.577676 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 08 18:48:32.577836 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:48:32.577752 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls podName:16722a0a-ba23-46c7-889f-f940d3083445 nodeName:}" failed. No retries permitted until 2026-09-08 18:48:33.077733329 +0000 UTC m=+871.586885756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls") pod "maas-api-dd87ddf4c-kp5lf" (UID: "16722a0a-ba23-46c7-889f-f940d3083445") : secret "maas-api-serving-cert" not found Sep 08 18:48:32.578189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.578170 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16722a0a-ba23-46c7-889f-f940d3083445-openshift-service-ca\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.579919 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.579894 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-metrics-certs\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.587881 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.587857 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvsz\" (UniqueName: \"kubernetes.io/projected/16722a0a-ba23-46c7-889f-f940d3083445-kube-api-access-hbvsz\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:32.920887 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.920854 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6qkth"] Sep 08 18:48:32.924356 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.924338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:32.926598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.926578 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 18:48:32.926716 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.926606 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g5jdp\"" Sep 08 18:48:32.933111 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:32.933090 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6qkth"] Sep 08 18:48:33.083014 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.082971 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:33.083199 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.083103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38351980-7f65-4fbb-b896-baecb32a11ca-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6qkth\" (UID: \"38351980-7f65-4fbb-b896-baecb32a11ca\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.083199 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.083133 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqb9j\" (UniqueName: \"kubernetes.io/projected/38351980-7f65-4fbb-b896-baecb32a11ca-kube-api-access-fqb9j\") pod \"payload-processing-5b54b7fbb7-6qkth\" (UID: \"38351980-7f65-4fbb-b896-baecb32a11ca\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.085701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.085668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls\") pod \"maas-api-dd87ddf4c-kp5lf\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:33.184545 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.184453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38351980-7f65-4fbb-b896-baecb32a11ca-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6qkth\" (UID: \"38351980-7f65-4fbb-b896-baecb32a11ca\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.184545 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.184490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqb9j\" (UniqueName: \"kubernetes.io/projected/38351980-7f65-4fbb-b896-baecb32a11ca-kube-api-access-fqb9j\") pod \"payload-processing-5b54b7fbb7-6qkth\" (UID: \"38351980-7f65-4fbb-b896-baecb32a11ca\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.185250 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.185224 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38351980-7f65-4fbb-b896-baecb32a11ca-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6qkth\" (UID: \"38351980-7f65-4fbb-b896-baecb32a11ca\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.192856 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.192831 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqb9j\" (UniqueName: \"kubernetes.io/projected/38351980-7f65-4fbb-b896-baecb32a11ca-kube-api-access-fqb9j\") pod \"payload-processing-5b54b7fbb7-6qkth\" (UID: \"38351980-7f65-4fbb-b896-baecb32a11ca\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.235946 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.235910 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:33.319724 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.319631 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:33.509669 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.509635 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6qkth"] Sep 08 18:48:33.511659 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:48:33.511624 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38351980_7f65_4fbb_b896_baecb32a11ca.slice/crio-50951fd6b39fdefc7065d8b22d5a8895e77ff47296e9ab265914cadbdba74ed5 WatchSource:0}: Error finding container 50951fd6b39fdefc7065d8b22d5a8895e77ff47296e9ab265914cadbdba74ed5: Status 404 returned error can't find the container with id 50951fd6b39fdefc7065d8b22d5a8895e77ff47296e9ab265914cadbdba74ed5 Sep 08 18:48:33.591913 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:33.591806 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf"] Sep 08 18:48:33.599476 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:48:33.599430 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16722a0a_ba23_46c7_889f_f940d3083445.slice/crio-873e9eac36328bffd2471558b824cf9a9b25c59ff837226e3cfd983fe16e52bf WatchSource:0}: Error finding container 873e9eac36328bffd2471558b824cf9a9b25c59ff837226e3cfd983fe16e52bf: Status 404 returned error can't find the container with id 873e9eac36328bffd2471558b824cf9a9b25c59ff837226e3cfd983fe16e52bf Sep 08 18:48:34.353904 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:34.353863 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" event={"ID":"16722a0a-ba23-46c7-889f-f940d3083445","Type":"ContainerStarted","Data":"873e9eac36328bffd2471558b824cf9a9b25c59ff837226e3cfd983fe16e52bf"} Sep 08 18:48:34.355467 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:34.355428 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" event={"ID":"38351980-7f65-4fbb-b896-baecb32a11ca","Type":"ContainerStarted","Data":"50951fd6b39fdefc7065d8b22d5a8895e77ff47296e9ab265914cadbdba74ed5"} Sep 08 18:48:36.368523 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:36.368430 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" event={"ID":"38351980-7f65-4fbb-b896-baecb32a11ca","Type":"ContainerStarted","Data":"0fdd069b3e2d8bd111e92fd600ae9b900f83c13bb9c3373bc184d306b4be3678"} Sep 08 18:48:36.368523 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:36.368516 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:36.370062 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:36.370036 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" event={"ID":"16722a0a-ba23-46c7-889f-f940d3083445","Type":"ContainerStarted","Data":"94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64"} Sep 08 18:48:36.370196 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:36.370167 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:36.384053 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:36.384009 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" podStartSLOduration=1.892914149 podStartE2EDuration="4.383995776s" totalImagesPullingTime="2.491081627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:36.382459424 +0000 UTC m=+874.891611866" watchObservedRunningTime="2026-09-08 18:48:36.383995776 +0000 UTC m=+874.893148211" Sep 08 18:48:36.398291 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:36.398245 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" podStartSLOduration=1.998734859 podStartE2EDuration="4.398232844s" totalImagesPullingTime="2.399497985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:36.396511181 +0000 UTC m=+874.905663621" watchObservedRunningTime="2026-09-08 18:48:36.398232844 +0000 UTC m=+874.907385285" Sep 08 18:48:40.942770 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:40.942733 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wf9s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 18:48:40.943157 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:40.942785 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podUID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 18:48:42.380128 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.380103 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 18:48:42.400098 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.400069 2557 generic.go:356] "Generic (PLEG): container finished" podID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerID="2f8b879fa1d9185067c4757d69ce68e9fea2870a5fdd64df6be9bca432a6f0cf" exitCode=0 Sep 08 18:48:42.400207 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.400139 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" event={"ID":"9dc57368-c1fe-40d7-bf89-4b4145a7497e","Type":"ContainerDied","Data":"2f8b879fa1d9185067c4757d69ce68e9fea2870a5fdd64df6be9bca432a6f0cf"} Sep 08 18:48:42.400507 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.400492 2557 scope.go:117] "RemoveContainer" containerID="2f8b879fa1d9185067c4757d69ce68e9fea2870a5fdd64df6be9bca432a6f0cf" Sep 08 18:48:42.401394 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.401371 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:48:42.401455 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.401443 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:48:42.401498 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:42.401488 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:48:43.407242 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:43.407205 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" event={"ID":"9dc57368-c1fe-40d7-bf89-4b4145a7497e","Type":"ContainerStarted","Data":"fd73bc59e4c8e020718b41f7b5b239bb5b0f43bea66c408fb822e0af84cfc6ed"} Sep 08 18:48:43.651040 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:43.651002 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:48:43.651257 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:43.651234 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wf9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 08 18:48:43.651304 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:43.651282 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podUID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 08 18:48:44.651372 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:44.651338 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wf9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 08 18:48:44.651754 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:44.651390 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podUID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 08 18:48:45.651426 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:45.651386 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wf9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 08 18:48:45.651822 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:45.651453 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podUID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 08 18:48:46.650827 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:46.650791 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wf9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 08 18:48:46.650997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:46.650849 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podUID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 08 18:48:47.376567 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:47.376536 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-6qkth" Sep 08 18:48:47.651334 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:47.651250 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wf9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 08 18:48:47.651334 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:47.651304 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" podUID="9dc57368-c1fe-40d7-bf89-4b4145a7497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 08 18:48:48.653682 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:48.653640 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:48:48.654124 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:48.654048 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:48:48.654600 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:48:48.654579 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wf9s5" Sep 08 18:49:28.275716 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.275682 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6"] Sep 08 18:49:28.279274 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.279257 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.281754 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.281729 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 18:49:28.282573 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.282553 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 18:49:28.282708 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.282573 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 08 18:49:28.282708 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.282596 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-htg2r\"" Sep 08 18:49:28.287280 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.287257 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6"] Sep 08 18:49:28.320839 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.320814 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.320943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.320846 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmqv\" (UniqueName: \"kubernetes.io/projected/f05c182b-7cd1-4088-88d2-a817d795618f-kube-api-access-wkmqv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.320943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.320867 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.321013 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.320955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.321013 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.320981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.321076 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.321008 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05c182b-7cd1-4088-88d2-a817d795618f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422056 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422181 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422063 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmqv\" (UniqueName: \"kubernetes.io/projected/f05c182b-7cd1-4088-88d2-a817d795618f-kube-api-access-wkmqv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422181 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422181 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422120 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422181 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422147 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422181 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05c182b-7cd1-4088-88d2-a817d795618f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422556 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422534 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422640 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422582 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.422726 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.422664 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.424358 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.424327 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05c182b-7cd1-4088-88d2-a817d795618f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.424465 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.424430 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05c182b-7cd1-4088-88d2-a817d795618f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.429143 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.429122 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmqv\" (UniqueName: \"kubernetes.io/projected/f05c182b-7cd1-4088-88d2-a817d795618f-kube-api-access-wkmqv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6\" (UID: \"f05c182b-7cd1-4088-88d2-a817d795618f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.591085 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.591014 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:28.861090 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.861067 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6"] Sep 08 18:49:28.863132 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:49:28.863098 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05c182b_7cd1_4088_88d2_a817d795618f.slice/crio-9809b650119cfea23bb4bcdaf316ab00f2d59b7511357de9a28065efaf236215 WatchSource:0}: Error finding container 9809b650119cfea23bb4bcdaf316ab00f2d59b7511357de9a28065efaf236215: Status 404 returned error can't find the container with id 9809b650119cfea23bb4bcdaf316ab00f2d59b7511357de9a28065efaf236215 Sep 08 18:49:28.864669 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:28.864625 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:49:29.652136 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:29.652095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" event={"ID":"f05c182b-7cd1-4088-88d2-a817d795618f","Type":"ContainerStarted","Data":"9809b650119cfea23bb4bcdaf316ab00f2d59b7511357de9a28065efaf236215"} Sep 08 18:49:30.658710 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:30.658677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" event={"ID":"f05c182b-7cd1-4088-88d2-a817d795618f","Type":"ContainerStarted","Data":"5229e2afe9adf92a6b19fe8280bc56c3445e7ed29453bf53e4038a1901babdab"} Sep 08 18:49:30.675556 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:30.675509 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" podStartSLOduration=1.665176663 podStartE2EDuration="2.675494622s" totalImagesPullingTime="1.010317959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:30.67291708 +0000 UTC m=+929.182069521" watchObservedRunningTime="2026-09-08 18:49:30.675494622 +0000 UTC m=+929.184647062" Sep 08 18:49:38.591329 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:38.591296 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:38.591803 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:38.591362 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:38.603491 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:38.603469 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:49:38.706075 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:49:38.706049 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6" Sep 08 18:51:24.506696 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.506662 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59cb7cd957-7cprf"] Sep 08 18:51:24.510069 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.510050 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.519803 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.519774 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59cb7cd957-7cprf"] Sep 08 18:51:24.588907 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.588878 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8982eaa-a448-41a3-9f44-cf35f46f7ba4-tls-cert\") pod \"authorino-59cb7cd957-7cprf\" (UID: \"f8982eaa-a448-41a3-9f44-cf35f46f7ba4\") " pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.589028 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.588927 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ltl\" (UniqueName: \"kubernetes.io/projected/f8982eaa-a448-41a3-9f44-cf35f46f7ba4-kube-api-access-99ltl\") pod \"authorino-59cb7cd957-7cprf\" (UID: \"f8982eaa-a448-41a3-9f44-cf35f46f7ba4\") " pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.689426 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.689402 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8982eaa-a448-41a3-9f44-cf35f46f7ba4-tls-cert\") pod \"authorino-59cb7cd957-7cprf\" (UID: \"f8982eaa-a448-41a3-9f44-cf35f46f7ba4\") " pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.689521 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.689447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ltl\" (UniqueName: \"kubernetes.io/projected/f8982eaa-a448-41a3-9f44-cf35f46f7ba4-kube-api-access-99ltl\") pod \"authorino-59cb7cd957-7cprf\" (UID: \"f8982eaa-a448-41a3-9f44-cf35f46f7ba4\") " pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.691737 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.691711 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8982eaa-a448-41a3-9f44-cf35f46f7ba4-tls-cert\") pod \"authorino-59cb7cd957-7cprf\" (UID: \"f8982eaa-a448-41a3-9f44-cf35f46f7ba4\") " pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.699531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.699508 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ltl\" (UniqueName: \"kubernetes.io/projected/f8982eaa-a448-41a3-9f44-cf35f46f7ba4-kube-api-access-99ltl\") pod \"authorino-59cb7cd957-7cprf\" (UID: \"f8982eaa-a448-41a3-9f44-cf35f46f7ba4\") " pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:24.826660 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:24.826591 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cb7cd957-7cprf" Sep 08 18:51:25.103584 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:25.103520 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59cb7cd957-7cprf"] Sep 08 18:51:25.106906 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:51:25.106872 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8982eaa_a448_41a3_9f44_cf35f46f7ba4.slice/crio-1bfdd11748c6161e079421d2b0d9b3abb7d22e9f1518c6c5f3e35bb480c8af2e WatchSource:0}: Error finding container 1bfdd11748c6161e079421d2b0d9b3abb7d22e9f1518c6c5f3e35bb480c8af2e: Status 404 returned error can't find the container with id 1bfdd11748c6161e079421d2b0d9b3abb7d22e9f1518c6c5f3e35bb480c8af2e Sep 08 18:51:25.213149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:25.213118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cb7cd957-7cprf" event={"ID":"f8982eaa-a448-41a3-9f44-cf35f46f7ba4","Type":"ContainerStarted","Data":"1bfdd11748c6161e079421d2b0d9b3abb7d22e9f1518c6c5f3e35bb480c8af2e"} Sep 08 18:51:26.222543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.222502 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cb7cd957-7cprf" event={"ID":"f8982eaa-a448-41a3-9f44-cf35f46f7ba4","Type":"ContainerStarted","Data":"1ee127e2e3a8fb491cdefb4ec08993f92d9d52b8c8a664d999ebf3fb961b8f90"} Sep 08 18:51:26.247492 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.247417 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59cb7cd957-7cprf" podStartSLOduration=1.56123086 podStartE2EDuration="2.247399509s" totalImagesPullingTime="686.168649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:26.245385362 +0000 UTC m=+1044.754537804" watchObservedRunningTime="2026-09-08 18:51:26.247399509 +0000 UTC m=+1044.756551949" Sep 08 18:51:26.277585 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.277546 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-598d6898d-8pgtt"] Sep 08 18:51:26.277787 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.277767 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-598d6898d-8pgtt" podUID="fec268dc-448a-4318-9676-151598e2cfb9" containerName="authorino" containerID="cri-o://9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091" gracePeriod=30 Sep 08 18:51:26.536431 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.536405 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:51:26.606983 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.606947 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fec268dc-448a-4318-9676-151598e2cfb9-tls-cert\") pod \"fec268dc-448a-4318-9676-151598e2cfb9\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " Sep 08 18:51:26.607282 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.607262 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nx9p\" (UniqueName: \"kubernetes.io/projected/fec268dc-448a-4318-9676-151598e2cfb9-kube-api-access-8nx9p\") pod \"fec268dc-448a-4318-9676-151598e2cfb9\" (UID: \"fec268dc-448a-4318-9676-151598e2cfb9\") " Sep 08 18:51:26.613226 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.613192 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec268dc-448a-4318-9676-151598e2cfb9-kube-api-access-8nx9p" (OuterVolumeSpecName: "kube-api-access-8nx9p") pod "fec268dc-448a-4318-9676-151598e2cfb9" (UID: "fec268dc-448a-4318-9676-151598e2cfb9"). InnerVolumeSpecName "kube-api-access-8nx9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:51:26.622271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.622244 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec268dc-448a-4318-9676-151598e2cfb9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fec268dc-448a-4318-9676-151598e2cfb9" (UID: "fec268dc-448a-4318-9676-151598e2cfb9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:51:26.709109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.709073 2557 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fec268dc-448a-4318-9676-151598e2cfb9-tls-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:51:26.709109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:26.709105 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nx9p\" (UniqueName: \"kubernetes.io/projected/fec268dc-448a-4318-9676-151598e2cfb9-kube-api-access-8nx9p\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:51:27.230338 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.230307 2557 generic.go:356] "Generic (PLEG): container finished" podID="fec268dc-448a-4318-9676-151598e2cfb9" containerID="9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091" exitCode=0 Sep 08 18:51:27.230730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.230355 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-598d6898d-8pgtt" Sep 08 18:51:27.230730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.230398 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-598d6898d-8pgtt" event={"ID":"fec268dc-448a-4318-9676-151598e2cfb9","Type":"ContainerDied","Data":"9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091"} Sep 08 18:51:27.230730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.230443 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-598d6898d-8pgtt" event={"ID":"fec268dc-448a-4318-9676-151598e2cfb9","Type":"ContainerDied","Data":"198ab6de044d83ec03d328899886fce7e55ab46c48bfa6d5a96ecac4ca07ed09"} Sep 08 18:51:27.230730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.230464 2557 scope.go:117] "RemoveContainer" containerID="9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091" Sep 08 18:51:27.240375 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.240345 2557 scope.go:117] "RemoveContainer" containerID="9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091" Sep 08 18:51:27.240627 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:51:27.240610 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091\": container with ID starting with 9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091 not found: ID does not exist" containerID="9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091" Sep 08 18:51:27.240730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.240635 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091"} err="failed to get container status \"9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091\": rpc error: code = NotFound desc = could not find container \"9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091\": container with ID starting with 9fd113569d07ecc7602ee9ae5e9bb5c8a952213150d48a44a8abd291a86e4091 not found: ID does not exist" Sep 08 18:51:27.252636 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.252611 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-598d6898d-8pgtt"] Sep 08 18:51:27.254321 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:27.254297 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-598d6898d-8pgtt"] Sep 08 18:51:28.027271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:28.027234 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec268dc-448a-4318-9676-151598e2cfb9" path="/var/lib/kubelet/pods/fec268dc-448a-4318-9676-151598e2cfb9/volumes" Sep 08 18:51:48.611867 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.611828 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq"] Sep 08 18:51:48.612453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.612433 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec268dc-448a-4318-9676-151598e2cfb9" containerName="authorino" Sep 08 18:51:48.612497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.612458 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec268dc-448a-4318-9676-151598e2cfb9" containerName="authorino" Sep 08 18:51:48.612588 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.612576 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec268dc-448a-4318-9676-151598e2cfb9" containerName="authorino" Sep 08 18:51:48.616291 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.616268 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.619024 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.619003 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52ba8b-openshift-default-dockercfg-zd8wq\"" Sep 08 18:51:48.638563 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.638514 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq"] Sep 08 18:51:48.783114 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783077 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-envoy\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783144 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-credential-socket\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91037de2-f80c-487c-9d82-f0ef84bf086b-istiod-ca-cert\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783249 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-socket\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-certs\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783310 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9p8k\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-kube-api-access-n9p8k\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783335 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-podinfo\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783373 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-data\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.783459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.783390 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-token\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884231 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884195 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-envoy\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884406 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884278 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-credential-socket\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884406 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91037de2-f80c-487c-9d82-f0ef84bf086b-istiod-ca-cert\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884406 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-socket\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-certs\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884449 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9p8k\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-kube-api-access-n9p8k\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884482 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-podinfo\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-data\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884553 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-token\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884894 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884736 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-credential-socket\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.884956 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.884922 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-certs\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.885217 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.885192 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91037de2-f80c-487c-9d82-f0ef84bf086b-istiod-ca-cert\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.885364 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.885197 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-data\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.885364 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.885283 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-socket\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.886828 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.886801 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-envoy\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.887094 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.887070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-podinfo\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.894897 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.894874 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9p8k\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-kube-api-access-n9p8k\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.895138 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.895119 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-token\") pod \"e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:48.933612 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:48.933570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:49.236080 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.236055 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq"] Sep 08 18:51:49.239165 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:51:49.239130 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91037de2_f80c_487c_9d82_f0ef84bf086b.slice/crio-a0214e8b8609562a0e4f0fe4731cc254d8a56a6ce50aa989b323549d5e28babe WatchSource:0}: Error finding container a0214e8b8609562a0e4f0fe4731cc254d8a56a6ce50aa989b323549d5e28babe: Status 404 returned error can't find the container with id a0214e8b8609562a0e4f0fe4731cc254d8a56a6ce50aa989b323549d5e28babe Sep 08 18:51:49.242125 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.242093 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:51:49.242231 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.242164 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:51:49.242231 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.242193 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:51:49.340448 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.340414 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" event={"ID":"91037de2-f80c-487c-9d82-f0ef84bf086b","Type":"ContainerStarted","Data":"1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69"} Sep 08 18:51:49.340634 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.340465 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" event={"ID":"91037de2-f80c-487c-9d82-f0ef84bf086b","Type":"ContainerStarted","Data":"a0214e8b8609562a0e4f0fe4731cc254d8a56a6ce50aa989b323549d5e28babe"} Sep 08 18:51:49.362027 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.361974 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podStartSLOduration=1.3619586799999999 podStartE2EDuration="1.36195868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:49.361231256 +0000 UTC m=+1067.870383698" watchObservedRunningTime="2026-09-08 18:51:49.36195868 +0000 UTC m=+1067.871111120" Sep 08 18:51:49.933878 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:49.933846 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:50.747328 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:50.747293 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-b06af2"] Sep 08 18:51:50.750868 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:50.750847 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:50.755255 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:50.755233 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\"" Sep 08 18:51:50.762071 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:50.762047 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-b06af2"] Sep 08 18:51:50.902600 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:50.902576 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvgp\" (UniqueName: \"kubernetes.io/projected/29eed290-b4fb-49e0-a88e-f048845726d8-kube-api-access-mmvgp\") pod \"test-curl-7704-b06af2\" (UID: \"29eed290-b4fb-49e0-a88e-f048845726d8\") " pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:50.939639 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:50.939615 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:51.003256 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.003202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvgp\" (UniqueName: \"kubernetes.io/projected/29eed290-b4fb-49e0-a88e-f048845726d8-kube-api-access-mmvgp\") pod \"test-curl-7704-b06af2\" (UID: \"29eed290-b4fb-49e0-a88e-f048845726d8\") " pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:51.012625 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.012602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvgp\" (UniqueName: \"kubernetes.io/projected/29eed290-b4fb-49e0-a88e-f048845726d8-kube-api-access-mmvgp\") pod \"test-curl-7704-b06af2\" (UID: \"29eed290-b4fb-49e0-a88e-f048845726d8\") " pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:51.060253 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.060224 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:51.334435 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.334403 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-b06af2"] Sep 08 18:51:51.335480 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:51:51.335454 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29eed290_b4fb_49e0_a88e_f048845726d8.slice/crio-9db1ad65b3de7438c278f9978f1e059e8c5eae1b2f1e1f51ddbb2e1ac9f0eb57 WatchSource:0}: Error finding container 9db1ad65b3de7438c278f9978f1e059e8c5eae1b2f1e1f51ddbb2e1ac9f0eb57: Status 404 returned error can't find the container with id 9db1ad65b3de7438c278f9978f1e059e8c5eae1b2f1e1f51ddbb2e1ac9f0eb57 Sep 08 18:51:51.352855 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.352824 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-b06af2" event={"ID":"29eed290-b4fb-49e0-a88e-f048845726d8","Type":"ContainerStarted","Data":"9db1ad65b3de7438c278f9978f1e059e8c5eae1b2f1e1f51ddbb2e1ac9f0eb57"} Sep 08 18:51:51.353250 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.353172 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:51.354127 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:51.354107 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:51:52.828164 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:52.828125 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw"] Sep 08 18:51:52.832866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:52.832842 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:52.838749 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:52.838721 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-52ba8b-serving-cert\"" Sep 08 18:51:52.854882 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:52.854861 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw"] Sep 08 18:51:53.020989 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.020547 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.020989 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.020606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-metrics-certs\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.022192 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.020636 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcz7p\" (UniqueName: \"kubernetes.io/projected/dacfd396-11ef-494a-b4c3-9d409c4e3c29-kube-api-access-rcz7p\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.022192 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.021344 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacfd396-11ef-494a-b4c3-9d409c4e3c29-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.122850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.122355 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.122850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.122405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-metrics-certs\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.122850 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:51:53.122471 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-serving-cert: secret "maas-api-e2e-inf-a-52ba8b-serving-cert" not found Sep 08 18:51:53.122850 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:51:53.122543 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls podName:dacfd396-11ef-494a-b4c3-9d409c4e3c29 nodeName:}" failed. No retries permitted until 2026-09-08 18:51:53.622521436 +0000 UTC m=+1072.131673869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls") pod "maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" (UID: "dacfd396-11ef-494a-b4c3-9d409c4e3c29") : secret "maas-api-e2e-inf-a-52ba8b-serving-cert" not found Sep 08 18:51:53.122850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.122689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcz7p\" (UniqueName: \"kubernetes.io/projected/dacfd396-11ef-494a-b4c3-9d409c4e3c29-kube-api-access-rcz7p\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.122850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.122769 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacfd396-11ef-494a-b4c3-9d409c4e3c29-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.123483 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.123457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacfd396-11ef-494a-b4c3-9d409c4e3c29-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.125754 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.125727 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-metrics-certs\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.163347 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.163289 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcz7p\" (UniqueName: \"kubernetes.io/projected/dacfd396-11ef-494a-b4c3-9d409c4e3c29-kube-api-access-rcz7p\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.627975 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.627929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.631033 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.631006 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls\") pod \"maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:53.746135 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:53.746092 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:54.088981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:54.088868 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw"] Sep 08 18:51:54.092603 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:51:54.092574 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacfd396_11ef_494a_b4c3_9d409c4e3c29.slice/crio-0c14156c0fabb23c65d4a8b3e7439f8701ac9e46922084399b7441c99d4baf0e WatchSource:0}: Error finding container 0c14156c0fabb23c65d4a8b3e7439f8701ac9e46922084399b7441c99d4baf0e: Status 404 returned error can't find the container with id 0c14156c0fabb23c65d4a8b3e7439f8701ac9e46922084399b7441c99d4baf0e Sep 08 18:51:54.374155 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:54.374078 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" event={"ID":"dacfd396-11ef-494a-b4c3-9d409c4e3c29","Type":"ContainerStarted","Data":"0c14156c0fabb23c65d4a8b3e7439f8701ac9e46922084399b7441c99d4baf0e"} Sep 08 18:51:54.375485 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:54.375460 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-b06af2" event={"ID":"29eed290-b4fb-49e0-a88e-f048845726d8","Type":"ContainerStarted","Data":"828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36"} Sep 08 18:51:54.402500 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:54.402461 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7704-b06af2" podStartSLOduration=1.887583801 podStartE2EDuration="4.402448133s" totalImagesPullingTime="2.514864332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:54.39976567 +0000 UTC m=+1072.908918121" watchObservedRunningTime="2026-09-08 18:51:54.402448133 +0000 UTC m=+1072.911600574" Sep 08 18:51:55.276773 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:55.276698 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-b06af2"] Sep 08 18:51:55.288562 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:55.288535 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-b06af2"] Sep 08 18:51:55.381909 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:55.381877 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" event={"ID":"dacfd396-11ef-494a-b4c3-9d409c4e3c29","Type":"ContainerStarted","Data":"dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142"} Sep 08 18:51:55.382093 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:51:55.382069 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7704-b06af2" container="test-curl-7704-b06af2" Sep 08 18:51:55.382171 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:55.382082 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:51:55.406903 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:55.406858 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" podStartSLOduration=2.802138745 podStartE2EDuration="3.406844587s" totalImagesPullingTime="604.705842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:55.405091481 +0000 UTC m=+1073.914243922" watchObservedRunningTime="2026-09-08 18:51:55.406844587 +0000 UTC m=+1073.915997028" Sep 08 18:51:56.386661 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:56.386614 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7704-b06af2" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" containerName="test-curl-7704-b06af2" containerID="cri-o://828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36" gracePeriod=2 Sep 08 18:51:58.620762 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:58.620726 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:58.773995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:58.773955 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmvgp\" (UniqueName: \"kubernetes.io/projected/29eed290-b4fb-49e0-a88e-f048845726d8-kube-api-access-mmvgp\") pod \"29eed290-b4fb-49e0-a88e-f048845726d8\" (UID: \"29eed290-b4fb-49e0-a88e-f048845726d8\") " Sep 08 18:51:58.776163 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:58.776121 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29eed290-b4fb-49e0-a88e-f048845726d8-kube-api-access-mmvgp" (OuterVolumeSpecName: "kube-api-access-mmvgp") pod "29eed290-b4fb-49e0-a88e-f048845726d8" (UID: "29eed290-b4fb-49e0-a88e-f048845726d8"). InnerVolumeSpecName "kube-api-access-mmvgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:51:58.875304 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:58.875274 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvgp\" (UniqueName: \"kubernetes.io/projected/29eed290-b4fb-49e0-a88e-f048845726d8-kube-api-access-mmvgp\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:51:59.405723 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.405678 2557 generic.go:356] "Generic (PLEG): container finished" podID="29eed290-b4fb-49e0-a88e-f048845726d8" containerID="828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36" exitCode=137 Sep 08 18:51:59.405723 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.405719 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:59.405959 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.405751 2557 scope.go:117] "RemoveContainer" containerID="828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36" Sep 08 18:51:59.414272 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.414252 2557 scope.go:117] "RemoveContainer" containerID="828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36" Sep 08 18:51:59.414555 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:51:59.414534 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36\": container with ID starting with 828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36 not found: ID does not exist" containerID="828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36" Sep 08 18:51:59.414609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.414566 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36"} err="failed to get container status \"828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36\": rpc error: code = NotFound desc = could not find container \"828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36\": container with ID starting with 828fd797bcb0422428c713385c36b0f0bc6c457d1ab8b0cd54e82492a3120d36 not found: ID does not exist" Sep 08 18:51:59.416515 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:51:59.416496 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-b06af2\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:51:59.999131 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.999078 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf"] Sep 08 18:51:59.999740 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.999719 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" containerName="test-curl-7704-b06af2" Sep 08 18:51:59.999856 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.999740 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" containerName="test-curl-7704-b06af2" Sep 08 18:51:59.999912 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:51:59.999862 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" containerName="test-curl-7704-b06af2" Sep 08 18:52:00.002861 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.002841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.004808 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:52:00.004783 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-b06af2\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" pod="opendatahub/test-curl-7704-b06af2" Sep 08 18:52:00.005222 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.005201 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-235f65-openshift-default-dockercfg-2vm4j\"" Sep 08 18:52:00.063370 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.063330 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29eed290-b4fb-49e0-a88e-f048845726d8" path="/var/lib/kubelet/pods/29eed290-b4fb-49e0-a88e-f048845726d8/volumes" Sep 08 18:52:00.063851 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.063826 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf"] Sep 08 18:52:00.085317 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085274 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-socket\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085426 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085320 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-podinfo\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085426 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085346 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hdk\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-kube-api-access-72hdk\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085426 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istiod-ca-cert\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085428 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-token\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-envoy\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085539 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-credential-socket\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-certs\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.085609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.085596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-data\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186442 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-token\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186623 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-envoy\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186623 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186484 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-credential-socket\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186623 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186510 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-certs\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186804 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-data\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186804 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-socket\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186804 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186766 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-podinfo\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.186804 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186792 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hdk\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-kube-api-access-72hdk\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.187011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istiod-ca-cert\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.187011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.186930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-credential-socket\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.187150 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.187126 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-data\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.187206 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.187137 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-socket\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.187381 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.187358 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-certs\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.187474 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.187458 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istiod-ca-cert\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.189192 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.189165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-envoy\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.189306 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.189247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-podinfo\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.197730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.197565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hdk\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-kube-api-access-72hdk\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.197730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.197622 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-token\") pod \"e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.313515 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.313433 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:00.634719 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.634323 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf"] Sep 08 18:52:00.645915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.642689 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:00.645915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.642776 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:00.645915 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:00.642823 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:01.392385 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:01.392355 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:52:01.420549 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:01.419772 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" event={"ID":"f9c087f8-0309-4ef1-a677-6a08f1e65db6","Type":"ContainerStarted","Data":"777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82"} Sep 08 18:52:01.420549 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:01.419832 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" event={"ID":"f9c087f8-0309-4ef1-a677-6a08f1e65db6","Type":"ContainerStarted","Data":"564817c9fde5ef92ccda17a4577f1348827f5819620aab27c074f194ec9b75c0"} Sep 08 18:52:01.450074 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:01.450019 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podStartSLOduration=2.449999566 podStartE2EDuration="2.449999566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:01.449121117 +0000 UTC m=+1079.958273573" watchObservedRunningTime="2026-09-08 18:52:01.449999566 +0000 UTC m=+1079.959152010" Sep 08 18:52:02.313760 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:02.313682 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:02.323106 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:02.323078 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:02.449582 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:02.448427 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:02.451578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:02.451555 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:05.108015 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.107982 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j"] Sep 08 18:52:05.116280 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.116251 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.131342 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.131310 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.131456 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.131391 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-metrics-certs\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.131456 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.131435 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133b257-8a21-48e5-a2a7-6192d52e25ca-openshift-service-ca\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.131572 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.131481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2w9s\" (UniqueName: \"kubernetes.io/projected/c133b257-8a21-48e5-a2a7-6192d52e25ca-kube-api-access-l2w9s\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.132420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.132353 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j"] Sep 08 18:52:05.132420 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.132381 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-235f65-serving-cert\"" Sep 08 18:52:05.232990 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.232953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-metrics-certs\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.233179 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.233013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133b257-8a21-48e5-a2a7-6192d52e25ca-openshift-service-ca\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.233179 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.233063 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2w9s\" (UniqueName: \"kubernetes.io/projected/c133b257-8a21-48e5-a2a7-6192d52e25ca-kube-api-access-l2w9s\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.233179 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.233116 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.233349 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:52:05.233227 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-235f65-serving-cert: secret "maas-api-e2e-autores-235f65-serving-cert" not found Sep 08 18:52:05.233349 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:52:05.233293 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls podName:c133b257-8a21-48e5-a2a7-6192d52e25ca nodeName:}" failed. No retries permitted until 2026-09-08 18:52:05.733270045 +0000 UTC m=+1084.242422465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls") pod "maas-api-e2e-autores-235f65-6d876c558d-hq56j" (UID: "c133b257-8a21-48e5-a2a7-6192d52e25ca") : secret "maas-api-e2e-autores-235f65-serving-cert" not found Sep 08 18:52:05.233925 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.233891 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133b257-8a21-48e5-a2a7-6192d52e25ca-openshift-service-ca\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.235532 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.235501 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-metrics-certs\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.242659 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.242613 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2w9s\" (UniqueName: \"kubernetes.io/projected/c133b257-8a21-48e5-a2a7-6192d52e25ca-kube-api-access-l2w9s\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.737710 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:05.737670 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:05.737915 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:52:05.737828 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-235f65-serving-cert: secret "maas-api-e2e-autores-235f65-serving-cert" not found Sep 08 18:52:05.737988 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:52:05.737915 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls podName:c133b257-8a21-48e5-a2a7-6192d52e25ca nodeName:}" failed. No retries permitted until 2026-09-08 18:52:06.737894553 +0000 UTC m=+1085.247046986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls") pod "maas-api-e2e-autores-235f65-6d876c558d-hq56j" (UID: "c133b257-8a21-48e5-a2a7-6192d52e25ca") : secret "maas-api-e2e-autores-235f65-serving-cert" not found Sep 08 18:52:06.749270 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:06.749218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:06.751608 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:06.751587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") pod \"maas-api-e2e-autores-235f65-6d876c558d-hq56j\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:06.933112 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:06.933070 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:07.214261 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:07.214238 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j"] Sep 08 18:52:07.218006 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:52:07.217977 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc133b257_8a21_48e5_a2a7_6192d52e25ca.slice/crio-c459491c26b6074d902eb72e341ba22a86430b29790fb823470074ce457aca65 WatchSource:0}: Error finding container c459491c26b6074d902eb72e341ba22a86430b29790fb823470074ce457aca65: Status 404 returned error can't find the container with id c459491c26b6074d902eb72e341ba22a86430b29790fb823470074ce457aca65 Sep 08 18:52:07.395814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:07.395767 2557 patch_prober.go:28] interesting pod/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 18:52:07.395969 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:07.395836 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 18:52:07.478710 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:07.478671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" event={"ID":"c133b257-8a21-48e5-a2a7-6192d52e25ca","Type":"ContainerStarted","Data":"c459491c26b6074d902eb72e341ba22a86430b29790fb823470074ce457aca65"} Sep 08 18:52:08.484611 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.484579 2557 generic.go:356] "Generic (PLEG): container finished" podID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerID="1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69" exitCode=0 Sep 08 18:52:08.485104 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.484677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" event={"ID":"91037de2-f80c-487c-9d82-f0ef84bf086b","Type":"ContainerDied","Data":"1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69"} Sep 08 18:52:08.485169 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.485134 2557 scope.go:117] "RemoveContainer" containerID="1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69" Sep 08 18:52:08.486314 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.486282 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:08.486432 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.486374 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:08.486432 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.486422 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:08.486551 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.486455 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" event={"ID":"c133b257-8a21-48e5-a2a7-6192d52e25ca","Type":"ContainerStarted","Data":"f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3"} Sep 08 18:52:08.486551 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.486521 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:08.520631 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:08.520583 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" podStartSLOduration=3.149161115 podStartE2EDuration="3.520563859s" totalImagesPullingTime="371.402744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:08.517604791 +0000 UTC m=+1087.026757233" watchObservedRunningTime="2026-09-08 18:52:08.520563859 +0000 UTC m=+1087.029716302" Sep 08 18:52:09.494372 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:09.494334 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" event={"ID":"91037de2-f80c-487c-9d82-f0ef84bf086b","Type":"ContainerStarted","Data":"9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc"} Sep 08 18:52:09.934692 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:09.934580 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:52:09.934875 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:09.934733 2557 patch_prober.go:28] interesting pod/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:09.934875 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:09.934778 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 18:52:10.934225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:10.934189 2557 patch_prober.go:28] interesting pod/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:10.934791 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:10.934757 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 18:52:11.934371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:11.934324 2557 patch_prober.go:28] interesting pod/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:11.934860 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:11.934399 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 18:52:12.934814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:12.934771 2557 patch_prober.go:28] interesting pod/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:12.935313 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:12.934835 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 18:52:13.934537 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:13.934477 2557 patch_prober.go:28] interesting pod/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:13.934750 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:13.934575 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 18:52:14.500371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:14.500336 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:52:14.937581 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:14.937507 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:52:14.937914 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:14.937887 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:52:14.938573 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:14.938550 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:52:17.324471 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.324436 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7"] Sep 08 18:52:17.329112 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.329085 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.332025 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.332002 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9c3074-openshift-default-dockercfg-lprpk\"" Sep 08 18:52:17.359200 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.359171 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7"] Sep 08 18:52:17.453590 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-data\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.453782 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-token\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.453782 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453669 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-certs\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.453782 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-podinfo\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.453928 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-socket\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.453928 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453884 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-credential-socket\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.453928 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453907 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlqh\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-kube-api-access-nwlqh\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.454059 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453960 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istiod-ca-cert\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.454059 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.453980 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-envoy\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554521 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554488 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-socket\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554521 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-credential-socket\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlqh\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-kube-api-access-nwlqh\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istiod-ca-cert\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-envoy\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-data\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-token\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.554765 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-certs\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.555058 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.554778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-podinfo\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.555058 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.555004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-socket\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.555255 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.555229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-data\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.555329 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.555309 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-certs\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.555423 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.555401 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-credential-socket\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.555518 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.555497 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istiod-ca-cert\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.557202 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.557174 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-podinfo\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.557350 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.557201 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-envoy\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.565670 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.565602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlqh\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-kube-api-access-nwlqh\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.566091 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.566071 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-token\") pod \"e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.640795 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.640714 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:17.930017 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.929991 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7"] Sep 08 18:52:17.933635 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:52:17.933604 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea58ca1_8151_4c6a_9aae_f87e5a63df08.slice/crio-3dc1f167cd450bc91666b3ce6fa65bf2a173ca23d45511471dd607e0a8bd4c52 WatchSource:0}: Error finding container 3dc1f167cd450bc91666b3ce6fa65bf2a173ca23d45511471dd607e0a8bd4c52: Status 404 returned error can't find the container with id 3dc1f167cd450bc91666b3ce6fa65bf2a173ca23d45511471dd607e0a8bd4c52 Sep 08 18:52:17.936788 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.936745 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:17.936939 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.936829 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:17.936939 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:17.936868 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:18.566669 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:18.565629 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" event={"ID":"6ea58ca1-8151-4c6a-9aae-f87e5a63df08","Type":"ContainerStarted","Data":"f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45"} Sep 08 18:52:18.566669 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:18.565724 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" event={"ID":"6ea58ca1-8151-4c6a-9aae-f87e5a63df08","Type":"ContainerStarted","Data":"3dc1f167cd450bc91666b3ce6fa65bf2a173ca23d45511471dd607e0a8bd4c52"} Sep 08 18:52:18.589668 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:18.588744 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podStartSLOduration=1.588723007 podStartE2EDuration="1.588723007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:18.587324294 +0000 UTC m=+1097.096476738" watchObservedRunningTime="2026-09-08 18:52:18.588723007 +0000 UTC m=+1097.097875450" Sep 08 18:52:18.646670 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:18.644748 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:19.647981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:19.647943 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:20.576519 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:20.576491 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:20.577474 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:20.577454 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:21.955324 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:21.955293 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5"] Sep 08 18:52:21.957695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:21.957672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:21.975434 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:21.975415 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9c3074-serving-cert\"" Sep 08 18:52:22.001718 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.001693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5"] Sep 08 18:52:22.007997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.007973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/480a7321-59f1-433e-b429-4bcd21fe57f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.008088 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.008052 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-metrics-certs\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.008162 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.008142 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wskn\" (UniqueName: \"kubernetes.io/projected/480a7321-59f1-433e-b429-4bcd21fe57f9-kube-api-access-5wskn\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.008201 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.008191 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.108850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.108819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wskn\" (UniqueName: \"kubernetes.io/projected/480a7321-59f1-433e-b429-4bcd21fe57f9-kube-api-access-5wskn\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.109090 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.108863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.109090 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.108930 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/480a7321-59f1-433e-b429-4bcd21fe57f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.109090 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.108964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-metrics-certs\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.110149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.110116 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/480a7321-59f1-433e-b429-4bcd21fe57f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.111628 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.111605 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.111759 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.111741 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-metrics-certs\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.135288 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.135260 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wskn\" (UniqueName: \"kubernetes.io/projected/480a7321-59f1-433e-b429-4bcd21fe57f9-kube-api-access-5wskn\") pod \"maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.269756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.269730 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:22.570366 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.570273 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5"] Sep 08 18:52:22.571071 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:52:22.571039 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480a7321_59f1_433e_b429_4bcd21fe57f9.slice/crio-676afaa7348f3d03e32d0415959f143c6ce1de3533a840f9d007d3cadd6f182d WatchSource:0}: Error finding container 676afaa7348f3d03e32d0415959f143c6ce1de3533a840f9d007d3cadd6f182d: Status 404 returned error can't find the container with id 676afaa7348f3d03e32d0415959f143c6ce1de3533a840f9d007d3cadd6f182d Sep 08 18:52:22.588307 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:22.588277 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" event={"ID":"480a7321-59f1-433e-b429-4bcd21fe57f9","Type":"ContainerStarted","Data":"676afaa7348f3d03e32d0415959f143c6ce1de3533a840f9d007d3cadd6f182d"} Sep 08 18:52:23.595908 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:23.595874 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" event={"ID":"480a7321-59f1-433e-b429-4bcd21fe57f9","Type":"ContainerStarted","Data":"653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c"} Sep 08 18:52:23.596261 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:23.595976 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:23.611534 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:23.611477 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" podStartSLOduration=2.27368871 podStartE2EDuration="2.611457548s" totalImagesPullingTime="337.768838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:23.610548416 +0000 UTC m=+1102.119700858" watchObservedRunningTime="2026-09-08 18:52:23.611457548 +0000 UTC m=+1102.120609989" Sep 08 18:52:27.632714 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:27.632673 2557 generic.go:356] "Generic (PLEG): container finished" podID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerID="777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82" exitCode=0 Sep 08 18:52:27.633188 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:27.632742 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" event={"ID":"f9c087f8-0309-4ef1-a677-6a08f1e65db6","Type":"ContainerDied","Data":"777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82"} Sep 08 18:52:27.633275 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:27.633259 2557 scope.go:117] "RemoveContainer" containerID="777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82" Sep 08 18:52:27.634561 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:27.634522 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:27.634703 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:27.634628 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:27.634782 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:27.634713 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:28.644860 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:28.644815 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" event={"ID":"f9c087f8-0309-4ef1-a677-6a08f1e65db6","Type":"ContainerStarted","Data":"a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053"} Sep 08 18:52:29.314508 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:29.314474 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:29.315850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:29.315639 2557 patch_prober.go:28] interesting pod/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 18:52:29.315850 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:29.315719 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 18:52:29.607762 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:29.607688 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:52:30.314849 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:30.314796 2557 patch_prober.go:28] interesting pod/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 18:52:30.315220 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:30.314881 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 18:52:31.314179 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:31.314142 2557 patch_prober.go:28] interesting pod/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 18:52:31.314351 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:31.314197 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 18:52:32.314358 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:32.314325 2557 patch_prober.go:28] interesting pod/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 18:52:32.314747 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:32.314379 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 18:52:32.448371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:32.448326 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:33.314388 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:33.314347 2557 patch_prober.go:28] interesting pod/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 18:52:33.314805 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:33.314403 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 18:52:34.319410 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:34.319374 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:34.320775 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:34.320748 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:52:36.447595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.446588 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:38036->10.133.0.52:15021: read: connection reset by peer" start-of-body= Sep 08 18:52:36.447595 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.446686 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:38036->10.133.0.52:15021: read: connection reset by peer" Sep 08 18:52:36.706039 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.705961 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerID="f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45" exitCode=0 Sep 08 18:52:36.706158 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.706035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" event={"ID":"6ea58ca1-8151-4c6a-9aae-f87e5a63df08","Type":"ContainerDied","Data":"f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45"} Sep 08 18:52:36.706472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.706455 2557 scope.go:117] "RemoveContainer" containerID="f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45" Sep 08 18:52:36.707517 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.707486 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:36.707624 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.707576 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:36.707711 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:36.707622 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:52:37.713453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:37.713406 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" event={"ID":"6ea58ca1-8151-4c6a-9aae-f87e5a63df08","Type":"ContainerStarted","Data":"773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db"} Sep 08 18:52:38.641247 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:38.641204 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:38.641607 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:38.641573 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 18:52:38.641733 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:38.641635 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 18:52:39.641896 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:39.641854 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 18:52:39.642878 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:39.642422 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 18:52:40.641632 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:40.641594 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 18:52:40.641960 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:40.641910 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 18:52:41.642042 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:41.642000 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 18:52:41.642526 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:41.642092 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 18:52:42.644669 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:42.641303 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 18:52:42.644669 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:42.641397 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 18:52:43.781905 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:43.781876 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:43.782304 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:43.782149 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:52:43.783049 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:52:43.783029 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:54:28.233141 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.233106 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j"] Sep 08 18:54:28.233826 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.233315 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" podUID="c133b257-8a21-48e5-a2a7-6192d52e25ca" containerName="maas-api" containerID="cri-o://f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3" gracePeriod=30 Sep 08 18:54:28.488110 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.488048 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:54:28.659879 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.659826 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") pod \"c133b257-8a21-48e5-a2a7-6192d52e25ca\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " Sep 08 18:54:28.659879 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.659879 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-metrics-certs\") pod \"c133b257-8a21-48e5-a2a7-6192d52e25ca\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " Sep 08 18:54:28.660102 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.659940 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133b257-8a21-48e5-a2a7-6192d52e25ca-openshift-service-ca\") pod \"c133b257-8a21-48e5-a2a7-6192d52e25ca\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " Sep 08 18:54:28.660102 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.660008 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2w9s\" (UniqueName: \"kubernetes.io/projected/c133b257-8a21-48e5-a2a7-6192d52e25ca-kube-api-access-l2w9s\") pod \"c133b257-8a21-48e5-a2a7-6192d52e25ca\" (UID: \"c133b257-8a21-48e5-a2a7-6192d52e25ca\") " Sep 08 18:54:28.661286 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.661017 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c133b257-8a21-48e5-a2a7-6192d52e25ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c133b257-8a21-48e5-a2a7-6192d52e25ca" (UID: "c133b257-8a21-48e5-a2a7-6192d52e25ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:28.664393 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.664248 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c133b257-8a21-48e5-a2a7-6192d52e25ca-kube-api-access-l2w9s" (OuterVolumeSpecName: "kube-api-access-l2w9s") pod "c133b257-8a21-48e5-a2a7-6192d52e25ca" (UID: "c133b257-8a21-48e5-a2a7-6192d52e25ca"). InnerVolumeSpecName "kube-api-access-l2w9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:28.664393 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.664354 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c133b257-8a21-48e5-a2a7-6192d52e25ca" (UID: "c133b257-8a21-48e5-a2a7-6192d52e25ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:54:28.664393 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.664367 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c133b257-8a21-48e5-a2a7-6192d52e25ca" (UID: "c133b257-8a21-48e5-a2a7-6192d52e25ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:54:28.761433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.761328 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133b257-8a21-48e5-a2a7-6192d52e25ca-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:28.761433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.761355 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2w9s\" (UniqueName: \"kubernetes.io/projected/c133b257-8a21-48e5-a2a7-6192d52e25ca-kube-api-access-l2w9s\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:28.761433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.761367 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:28.761433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:28.761376 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c133b257-8a21-48e5-a2a7-6192d52e25ca-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:29.347293 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.347256 2557 generic.go:356] "Generic (PLEG): container finished" podID="c133b257-8a21-48e5-a2a7-6192d52e25ca" containerID="f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3" exitCode=0 Sep 08 18:54:29.347783 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.347312 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" Sep 08 18:54:29.347783 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.347342 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" event={"ID":"c133b257-8a21-48e5-a2a7-6192d52e25ca","Type":"ContainerDied","Data":"f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3"} Sep 08 18:54:29.347783 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.347404 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j" event={"ID":"c133b257-8a21-48e5-a2a7-6192d52e25ca","Type":"ContainerDied","Data":"c459491c26b6074d902eb72e341ba22a86430b29790fb823470074ce457aca65"} Sep 08 18:54:29.347783 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.347437 2557 scope.go:117] "RemoveContainer" containerID="f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3" Sep 08 18:54:29.359030 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.358562 2557 scope.go:117] "RemoveContainer" containerID="f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3" Sep 08 18:54:29.359030 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:54:29.358883 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3\": container with ID starting with f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3 not found: ID does not exist" containerID="f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3" Sep 08 18:54:29.359030 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.358938 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3"} err="failed to get container status \"f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3\": rpc error: code = NotFound desc = could not find container \"f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3\": container with ID starting with f3b69420d04d2e8b3953f7ad48a5472fa60bd32fe0b5e6b3ba3d83fed9a384d3 not found: ID does not exist" Sep 08 18:54:29.382238 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.382208 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j"] Sep 08 18:54:29.387531 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:29.387505 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-235f65-6d876c558d-hq56j"] Sep 08 18:54:30.027433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:30.027402 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c133b257-8a21-48e5-a2a7-6192d52e25ca" path="/var/lib/kubelet/pods/c133b257-8a21-48e5-a2a7-6192d52e25ca/volumes" Sep 08 18:54:35.696466 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:35.696435 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf"] Sep 08 18:54:35.696960 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:35.696672 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" containerID="cri-o://a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053" gracePeriod=30 Sep 08 18:54:40.953779 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:40.953749 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:54:41.065617 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.065590 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-podinfo\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066317 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066288 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-token\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066440 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066327 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-envoy\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066440 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066378 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-credential-socket\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066440 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066411 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-certs\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066496 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72hdk\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-kube-api-access-72hdk\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066523 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istiod-ca-cert\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066554 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-data\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.066799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066669 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:41.067026 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066900 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-socket\") pod \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\" (UID: \"f9c087f8-0309-4ef1-a677-6a08f1e65db6\") " Sep 08 18:54:41.067026 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.066935 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:41.067143 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.067084 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:41.067271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.067212 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:41.067271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.067237 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.067271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.067254 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.067271 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.067269 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.067525 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.067413 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:41.069358 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.069241 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:54:41.069358 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.069330 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-kube-api-access-72hdk" (OuterVolumeSpecName: "kube-api-access-72hdk") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "kube-api-access-72hdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:41.069544 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.069368 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:41.069668 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.069623 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9c087f8-0309-4ef1-a677-6a08f1e65db6" (UID: "f9c087f8-0309-4ef1-a677-6a08f1e65db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:41.168178 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.168153 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.168178 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.168177 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.168335 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.168190 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.168335 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.168203 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.168335 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.168215 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c087f8-0309-4ef1-a677-6a08f1e65db6-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.168335 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.168227 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hdk\" (UniqueName: \"kubernetes.io/projected/f9c087f8-0309-4ef1-a677-6a08f1e65db6-kube-api-access-72hdk\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:54:41.413379 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.413298 2557 generic.go:356] "Generic (PLEG): container finished" podID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerID="a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053" exitCode=0 Sep 08 18:54:41.413379 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.413353 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" Sep 08 18:54:41.413565 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.413372 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" event={"ID":"f9c087f8-0309-4ef1-a677-6a08f1e65db6","Type":"ContainerDied","Data":"a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053"} Sep 08 18:54:41.413565 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.413424 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf" event={"ID":"f9c087f8-0309-4ef1-a677-6a08f1e65db6","Type":"ContainerDied","Data":"564817c9fde5ef92ccda17a4577f1348827f5819620aab27c074f194ec9b75c0"} Sep 08 18:54:41.413565 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.413447 2557 scope.go:117] "RemoveContainer" containerID="a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053" Sep 08 18:54:41.424321 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.424305 2557 scope.go:117] "RemoveContainer" containerID="777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82" Sep 08 18:54:41.432081 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.432026 2557 scope.go:117] "RemoveContainer" containerID="a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053" Sep 08 18:54:41.432339 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:54:41.432319 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053\": container with ID starting with a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053 not found: ID does not exist" containerID="a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053" Sep 08 18:54:41.432427 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.432352 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053"} err="failed to get container status \"a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053\": rpc error: code = NotFound desc = could not find container \"a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053\": container with ID starting with a7e4bb1e418b941f38b84fcde6cb260724ecc73a3f8ab853f66c1b6ef5d03053 not found: ID does not exist" Sep 08 18:54:41.432427 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.432375 2557 scope.go:117] "RemoveContainer" containerID="777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82" Sep 08 18:54:41.432706 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:54:41.432688 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82\": container with ID starting with 777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82 not found: ID does not exist" containerID="777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82" Sep 08 18:54:41.432757 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.432712 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82"} err="failed to get container status \"777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82\": rpc error: code = NotFound desc = could not find container \"777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82\": container with ID starting with 777aa6aa4100c044541d15f1867f2a9f2df7ef476b719ad8132db9a7b245eb82 not found: ID does not exist" Sep 08 18:54:41.448200 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.448169 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf"] Sep 08 18:54:41.455866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:41.455835 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-235f65-openshift-default-79f865dcfb-rmcgf"] Sep 08 18:54:42.027361 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:54:42.027325 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" path="/var/lib/kubelet/pods/f9c087f8-0309-4ef1-a677-6a08f1e65db6/volumes" Sep 08 18:55:41.261755 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.261722 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr"] Sep 08 18:55:41.262367 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262344 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" Sep 08 18:55:41.262451 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262369 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" Sep 08 18:55:41.262451 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262421 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c133b257-8a21-48e5-a2a7-6192d52e25ca" containerName="maas-api" Sep 08 18:55:41.262451 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262430 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c133b257-8a21-48e5-a2a7-6192d52e25ca" containerName="maas-api" Sep 08 18:55:41.262604 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262527 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" Sep 08 18:55:41.262604 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262539 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" Sep 08 18:55:41.262604 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.262557 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c133b257-8a21-48e5-a2a7-6192d52e25ca" containerName="maas-api" Sep 08 18:55:41.266017 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.265995 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.268403 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.268383 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-3b2d3e-openshift-default-dockercfg-2z28w\"" Sep 08 18:55:41.277135 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.277116 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr"] Sep 08 18:55:41.380039 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.379998 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fkxd\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-kube-api-access-6fkxd\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-data\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380086 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-socket\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380110 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istiod-ca-cert\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380143 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-podinfo\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380240 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-credential-socket\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380536 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380282 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-token\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380536 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380367 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-certs\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.380536 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.380415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-envoy\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481060 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481027 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-certs\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481060 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-envoy\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481267 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fkxd\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-kube-api-access-6fkxd\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481267 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481120 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-data\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481267 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481155 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-socket\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481267 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istiod-ca-cert\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481267 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-podinfo\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481267 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-credential-socket\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481591 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-token\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481694 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481638 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-socket\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481694 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481682 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-credential-socket\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481828 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481807 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-certs\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481898 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481883 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-data\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.481975 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.481950 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istiod-ca-cert\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.483901 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.483878 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-envoy\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.484047 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.484024 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-podinfo\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.489307 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.489276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-token\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.489622 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.489605 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fkxd\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-kube-api-access-6fkxd\") pod \"e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.577134 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.577067 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:41.857401 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.857379 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr"] Sep 08 18:55:41.859750 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:55:41.859725 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424cbd03_58dd_4dd8_b2c5_1f7d72be80ee.slice/crio-3f2e8fda49e40fde17121604b57bdf2073cd80687c092f56f88bb96c73585b78 WatchSource:0}: Error finding container 3f2e8fda49e40fde17121604b57bdf2073cd80687c092f56f88bb96c73585b78: Status 404 returned error can't find the container with id 3f2e8fda49e40fde17121604b57bdf2073cd80687c092f56f88bb96c73585b78 Sep 08 18:55:41.861937 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.861906 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:55:41.862032 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.861972 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:55:41.862032 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:41.862001 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:55:42.742133 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:42.742099 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" event={"ID":"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee","Type":"ContainerStarted","Data":"83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c"} Sep 08 18:55:42.742498 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:42.742154 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" event={"ID":"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee","Type":"ContainerStarted","Data":"3f2e8fda49e40fde17121604b57bdf2073cd80687c092f56f88bb96c73585b78"} Sep 08 18:55:42.766950 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:42.766904 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" podStartSLOduration=1.766890324 podStartE2EDuration="1.766890324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:55:42.765435204 +0000 UTC m=+1301.274587636" watchObservedRunningTime="2026-09-08 18:55:42.766890324 +0000 UTC m=+1301.276042765" Sep 08 18:55:43.577404 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:43.577370 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:43.582233 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:43.582206 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:43.748000 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:43.747964 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:55:43.748880 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:55:43.748857 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:56:06.152441 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:06.152410 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr"] Sep 08 18:56:06.152927 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:06.152618 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" podUID="424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" containerName="istio-proxy" containerID="cri-o://83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c" gracePeriod=30 Sep 08 18:56:11.404931 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.404906 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:56:11.546662 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546614 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-credential-socket\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.546824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546675 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-podinfo\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.546824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546710 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-token\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.546824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546726 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-envoy\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.546824 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546765 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-certs\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.547057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546852 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:11.547057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546910 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-data\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.547057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546944 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fkxd\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-kube-api-access-6fkxd\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.547057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.546986 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-socket\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.547057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547011 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istiod-ca-cert\") pod \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\" (UID: \"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee\") " Sep 08 18:56:11.547371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547085 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:11.547371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547288 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:11.547495 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547380 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.547495 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547400 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.547495 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547414 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.547690 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547486 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:11.547690 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.547575 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:11.549190 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.549156 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:56:11.549302 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.549192 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-kube-api-access-6fkxd" (OuterVolumeSpecName: "kube-api-access-6fkxd") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "kube-api-access-6fkxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:11.549302 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.549263 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:11.549516 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.549489 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" (UID: "424cbd03-58dd-4dd8-b2c5-1f7d72be80ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:11.648233 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.648211 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.648233 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.648233 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.648387 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.648243 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.648387 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.648251 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fkxd\" (UniqueName: \"kubernetes.io/projected/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-kube-api-access-6fkxd\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.648387 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.648260 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.648387 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.648268 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:11.897505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.897426 2557 generic.go:356] "Generic (PLEG): container finished" podID="424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" containerID="83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c" exitCode=0 Sep 08 18:56:11.897505 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.897477 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" Sep 08 18:56:11.897799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.897508 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" event={"ID":"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee","Type":"ContainerDied","Data":"83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c"} Sep 08 18:56:11.897799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.897568 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr" event={"ID":"424cbd03-58dd-4dd8-b2c5-1f7d72be80ee","Type":"ContainerDied","Data":"3f2e8fda49e40fde17121604b57bdf2073cd80687c092f56f88bb96c73585b78"} Sep 08 18:56:11.897799 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.897599 2557 scope.go:117] "RemoveContainer" containerID="83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c" Sep 08 18:56:11.906979 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.906962 2557 scope.go:117] "RemoveContainer" containerID="83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c" Sep 08 18:56:11.907245 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:56:11.907226 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c\": container with ID starting with 83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c not found: ID does not exist" containerID="83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c" Sep 08 18:56:11.907295 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.907253 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c"} err="failed to get container status \"83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c\": rpc error: code = NotFound desc = could not find container \"83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c\": container with ID starting with 83b2a446601c7a3b71e2ff540a31c6787e7a65d9d9bfeaac78a0e8f2506e6b4c not found: ID does not exist" Sep 08 18:56:11.933285 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.933263 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr"] Sep 08 18:56:11.944019 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:11.943995 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-3b2d3e-openshift-default-5d57487bcd-ljvdr"] Sep 08 18:56:12.027464 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:12.027437 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" path="/var/lib/kubelet/pods/424cbd03-58dd-4dd8-b2c5-1f7d72be80ee/volumes" Sep 08 18:56:30.188199 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.188103 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw"] Sep 08 18:56:30.188673 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.188330 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" podUID="dacfd396-11ef-494a-b4c3-9d409c4e3c29" containerName="maas-api" containerID="cri-o://dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142" gracePeriod=30 Sep 08 18:56:30.635130 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.635097 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:56:30.707535 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.707504 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacfd396-11ef-494a-b4c3-9d409c4e3c29-openshift-service-ca\") pod \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " Sep 08 18:56:30.707695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.707547 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-metrics-certs\") pod \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " Sep 08 18:56:30.707695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.707615 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcz7p\" (UniqueName: \"kubernetes.io/projected/dacfd396-11ef-494a-b4c3-9d409c4e3c29-kube-api-access-rcz7p\") pod \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " Sep 08 18:56:30.707781 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.707714 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls\") pod \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\" (UID: \"dacfd396-11ef-494a-b4c3-9d409c4e3c29\") " Sep 08 18:56:30.707932 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.707908 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacfd396-11ef-494a-b4c3-9d409c4e3c29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dacfd396-11ef-494a-b4c3-9d409c4e3c29" (UID: "dacfd396-11ef-494a-b4c3-9d409c4e3c29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:30.709692 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.709641 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dacfd396-11ef-494a-b4c3-9d409c4e3c29" (UID: "dacfd396-11ef-494a-b4c3-9d409c4e3c29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:56:30.709792 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.709690 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dacfd396-11ef-494a-b4c3-9d409c4e3c29" (UID: "dacfd396-11ef-494a-b4c3-9d409c4e3c29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:56:30.709792 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.709775 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacfd396-11ef-494a-b4c3-9d409c4e3c29-kube-api-access-rcz7p" (OuterVolumeSpecName: "kube-api-access-rcz7p") pod "dacfd396-11ef-494a-b4c3-9d409c4e3c29" (UID: "dacfd396-11ef-494a-b4c3-9d409c4e3c29"). InnerVolumeSpecName "kube-api-access-rcz7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:30.808514 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.808488 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcz7p\" (UniqueName: \"kubernetes.io/projected/dacfd396-11ef-494a-b4c3-9d409c4e3c29-kube-api-access-rcz7p\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.808514 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.808512 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.808688 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.808524 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacfd396-11ef-494a-b4c3-9d409c4e3c29-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.808688 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:30.808532 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacfd396-11ef-494a-b4c3-9d409c4e3c29-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:31.008307 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.008277 2557 generic.go:356] "Generic (PLEG): container finished" podID="dacfd396-11ef-494a-b4c3-9d409c4e3c29" containerID="dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142" exitCode=0 Sep 08 18:56:31.008421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.008324 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" Sep 08 18:56:31.008421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.008348 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" event={"ID":"dacfd396-11ef-494a-b4c3-9d409c4e3c29","Type":"ContainerDied","Data":"dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142"} Sep 08 18:56:31.008421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.008389 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw" event={"ID":"dacfd396-11ef-494a-b4c3-9d409c4e3c29","Type":"ContainerDied","Data":"0c14156c0fabb23c65d4a8b3e7439f8701ac9e46922084399b7441c99d4baf0e"} Sep 08 18:56:31.008421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.008409 2557 scope.go:117] "RemoveContainer" containerID="dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142" Sep 08 18:56:31.021666 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.021624 2557 scope.go:117] "RemoveContainer" containerID="dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142" Sep 08 18:56:31.021967 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:56:31.021939 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142\": container with ID starting with dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142 not found: ID does not exist" containerID="dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142" Sep 08 18:56:31.022066 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.021976 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142"} err="failed to get container status \"dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142\": rpc error: code = NotFound desc = could not find container \"dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142\": container with ID starting with dd89267482ffcbe5dab1e581b85e7cea66e2b3d8d8772da837a400bdfa3ac142 not found: ID does not exist" Sep 08 18:56:31.031340 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.031317 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw"] Sep 08 18:56:31.037066 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:31.037034 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52ba8b-85f89bf9fc-hj2fw"] Sep 08 18:56:32.027522 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:32.027490 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dacfd396-11ef-494a-b4c3-9d409c4e3c29" path="/var/lib/kubelet/pods/dacfd396-11ef-494a-b4c3-9d409c4e3c29/volumes" Sep 08 18:56:37.237890 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:37.237856 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq"] Sep 08 18:56:37.238288 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:37.238099 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" containerID="cri-o://9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc" gracePeriod=30 Sep 08 18:56:42.484083 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.484058 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:56:42.612734 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612659 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-podinfo\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612734 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612732 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-token\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612767 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-credential-socket\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612797 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9p8k\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-kube-api-access-n9p8k\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612836 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-data\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612879 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-certs\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612905 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-socket\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.612943 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612940 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-envoy\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.613259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.612979 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91037de2-f80c-487c-9d82-f0ef84bf086b-istiod-ca-cert\") pod \"91037de2-f80c-487c-9d82-f0ef84bf086b\" (UID: \"91037de2-f80c-487c-9d82-f0ef84bf086b\") " Sep 08 18:56:42.613259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.613103 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:42.613259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.613115 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-data" (OuterVolumeSpecName: "istio-data") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:42.613421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.613305 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.613421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.613309 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:42.613421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.613326 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.613924 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.613894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:42.614057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.614003 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91037de2-f80c-487c-9d82-f0ef84bf086b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:42.615143 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.615120 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-kube-api-access-n9p8k" (OuterVolumeSpecName: "kube-api-access-n9p8k") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "kube-api-access-n9p8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:42.615297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.615279 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:56:42.615361 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.615325 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-token" (OuterVolumeSpecName: "istio-token") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:42.615634 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.615613 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91037de2-f80c-487c-9d82-f0ef84bf086b" (UID: "91037de2-f80c-487c-9d82-f0ef84bf086b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:56:42.714130 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714107 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.714130 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714128 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9p8k\" (UniqueName: \"kubernetes.io/projected/91037de2-f80c-487c-9d82-f0ef84bf086b-kube-api-access-n9p8k\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.714266 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714137 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.714266 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714146 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.714266 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714153 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.714266 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714161 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91037de2-f80c-487c-9d82-f0ef84bf086b-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:42.714266 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:42.714168 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91037de2-f80c-487c-9d82-f0ef84bf086b-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:43.072497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.072464 2557 generic.go:356] "Generic (PLEG): container finished" podID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerID="9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc" exitCode=0 Sep 08 18:56:43.072624 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.072499 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" Sep 08 18:56:43.072624 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.072552 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" event={"ID":"91037de2-f80c-487c-9d82-f0ef84bf086b","Type":"ContainerDied","Data":"9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc"} Sep 08 18:56:43.072624 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.072598 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq" event={"ID":"91037de2-f80c-487c-9d82-f0ef84bf086b","Type":"ContainerDied","Data":"a0214e8b8609562a0e4f0fe4731cc254d8a56a6ce50aa989b323549d5e28babe"} Sep 08 18:56:43.072624 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.072622 2557 scope.go:117] "RemoveContainer" containerID="9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc" Sep 08 18:56:43.081848 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.081830 2557 scope.go:117] "RemoveContainer" containerID="1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69" Sep 08 18:56:43.089479 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.089462 2557 scope.go:117] "RemoveContainer" containerID="9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc" Sep 08 18:56:43.089747 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:56:43.089730 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc\": container with ID starting with 9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc not found: ID does not exist" containerID="9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc" Sep 08 18:56:43.089816 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.089752 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc"} err="failed to get container status \"9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc\": rpc error: code = NotFound desc = could not find container \"9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc\": container with ID starting with 9588308229db9dcb41e95e11280f28f32ed149010c222a2d9f4fe8edcc52e7cc not found: ID does not exist" Sep 08 18:56:43.089816 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.089767 2557 scope.go:117] "RemoveContainer" containerID="1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69" Sep 08 18:56:43.090015 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:56:43.089998 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69\": container with ID starting with 1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69 not found: ID does not exist" containerID="1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69" Sep 08 18:56:43.090060 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.090023 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69"} err="failed to get container status \"1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69\": rpc error: code = NotFound desc = could not find container \"1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69\": container with ID starting with 1d902d3a38027c78dc0c9d2c95842ee8c5fb8a385fba91d413944d2766592a69 not found: ID does not exist" Sep 08 18:56:43.092995 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.092976 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq"] Sep 08 18:56:43.096570 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:43.096549 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52ba8b-openshift-default-65dddd68cd-cczcq"] Sep 08 18:56:44.026554 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:44.026512 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" path="/var/lib/kubelet/pods/91037de2-f80c-487c-9d82-f0ef84bf086b/volumes" Sep 08 18:56:48.324821 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.324788 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5"] Sep 08 18:56:48.325579 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.325533 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" podUID="480a7321-59f1-433e-b429-4bcd21fe57f9" containerName="maas-api" containerID="cri-o://653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c" gracePeriod=30 Sep 08 18:56:48.778879 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.778852 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:56:48.863830 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.863759 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-maas-api-tls\") pod \"480a7321-59f1-433e-b429-4bcd21fe57f9\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " Sep 08 18:56:48.863830 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.863799 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wskn\" (UniqueName: \"kubernetes.io/projected/480a7321-59f1-433e-b429-4bcd21fe57f9-kube-api-access-5wskn\") pod \"480a7321-59f1-433e-b429-4bcd21fe57f9\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " Sep 08 18:56:48.864044 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.863888 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/480a7321-59f1-433e-b429-4bcd21fe57f9-openshift-service-ca\") pod \"480a7321-59f1-433e-b429-4bcd21fe57f9\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " Sep 08 18:56:48.864044 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.863905 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-metrics-certs\") pod \"480a7321-59f1-433e-b429-4bcd21fe57f9\" (UID: \"480a7321-59f1-433e-b429-4bcd21fe57f9\") " Sep 08 18:56:48.864260 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.864231 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480a7321-59f1-433e-b429-4bcd21fe57f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "480a7321-59f1-433e-b429-4bcd21fe57f9" (UID: "480a7321-59f1-433e-b429-4bcd21fe57f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:48.865917 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.865886 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "480a7321-59f1-433e-b429-4bcd21fe57f9" (UID: "480a7321-59f1-433e-b429-4bcd21fe57f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:56:48.866030 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.865980 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480a7321-59f1-433e-b429-4bcd21fe57f9-kube-api-access-5wskn" (OuterVolumeSpecName: "kube-api-access-5wskn") pod "480a7321-59f1-433e-b429-4bcd21fe57f9" (UID: "480a7321-59f1-433e-b429-4bcd21fe57f9"). InnerVolumeSpecName "kube-api-access-5wskn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:48.866094 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.866077 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "480a7321-59f1-433e-b429-4bcd21fe57f9" (UID: "480a7321-59f1-433e-b429-4bcd21fe57f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:56:48.965314 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.965280 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/480a7321-59f1-433e-b429-4bcd21fe57f9-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.965314 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.965314 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.965476 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.965329 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/480a7321-59f1-433e-b429-4bcd21fe57f9-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.965476 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:48.965342 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wskn\" (UniqueName: \"kubernetes.io/projected/480a7321-59f1-433e-b429-4bcd21fe57f9-kube-api-access-5wskn\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:56:49.103582 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.103549 2557 generic.go:356] "Generic (PLEG): container finished" podID="480a7321-59f1-433e-b429-4bcd21fe57f9" containerID="653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c" exitCode=0 Sep 08 18:56:49.103756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.103608 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" Sep 08 18:56:49.103756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.103641 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" event={"ID":"480a7321-59f1-433e-b429-4bcd21fe57f9","Type":"ContainerDied","Data":"653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c"} Sep 08 18:56:49.103756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.103721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5" event={"ID":"480a7321-59f1-433e-b429-4bcd21fe57f9","Type":"ContainerDied","Data":"676afaa7348f3d03e32d0415959f143c6ce1de3533a840f9d007d3cadd6f182d"} Sep 08 18:56:49.103756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.103754 2557 scope.go:117] "RemoveContainer" containerID="653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c" Sep 08 18:56:49.113708 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.113692 2557 scope.go:117] "RemoveContainer" containerID="653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c" Sep 08 18:56:49.114024 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:56:49.113972 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c\": container with ID starting with 653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c not found: ID does not exist" containerID="653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c" Sep 08 18:56:49.114024 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.114005 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c"} err="failed to get container status \"653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c\": rpc error: code = NotFound desc = could not find container \"653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c\": container with ID starting with 653ca9a793d518d686c439aff68f1975ec54b4a347df18c11fea425460e3367c not found: ID does not exist" Sep 08 18:56:49.125723 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.125689 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5"] Sep 08 18:56:49.129205 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:49.129186 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3074-6bdd7fdd8-9krg5"] Sep 08 18:56:50.026017 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:50.025981 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480a7321-59f1-433e-b429-4bcd21fe57f9" path="/var/lib/kubelet/pods/480a7321-59f1-433e-b429-4bcd21fe57f9/volumes" Sep 08 18:56:55.293557 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:55.293524 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7"] Sep 08 18:56:55.294032 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:55.293803 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" containerID="cri-o://773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db" gracePeriod=30 Sep 08 18:56:56.214052 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214013 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79"] Sep 08 18:56:56.214414 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214401 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" Sep 08 18:56:56.214468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214416 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c087f8-0309-4ef1-a677-6a08f1e65db6" containerName="istio-proxy" Sep 08 18:56:56.214468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214436 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacfd396-11ef-494a-b4c3-9d409c4e3c29" containerName="maas-api" Sep 08 18:56:56.214468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214443 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacfd396-11ef-494a-b4c3-9d409c4e3c29" containerName="maas-api" Sep 08 18:56:56.214468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214451 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" Sep 08 18:56:56.214468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214458 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214472 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214480 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214489 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214497 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214513 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480a7321-59f1-433e-b429-4bcd21fe57f9" containerName="maas-api" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214520 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="480a7321-59f1-433e-b429-4bcd21fe57f9" containerName="maas-api" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214619 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214629 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="91037de2-f80c-487c-9d82-f0ef84bf086b" containerName="istio-proxy" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214640 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacfd396-11ef-494a-b4c3-9d409c4e3c29" containerName="maas-api" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214662 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="480a7321-59f1-433e-b429-4bcd21fe57f9" containerName="maas-api" Sep 08 18:56:56.214701 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.214670 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="424cbd03-58dd-4dd8-b2c5-1f7d72be80ee" containerName="istio-proxy" Sep 08 18:56:56.219473 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.219449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.221876 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.221857 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a4001b-openshift-default-dockercfg-2qgqw\"" Sep 08 18:56:56.230917 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.230895 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79"] Sep 08 18:56:56.322608 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322569 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-certs\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322623 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-socket\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322689 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-credential-socket\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322745 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlq8\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-kube-api-access-ddlq8\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322781 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-data\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322824 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e0bf24-3c2f-4161-a254-8c889f0630aa-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-envoy\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322889 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-podinfo\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.323082 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.322995 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-token\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.423727 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.423698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlq8\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-kube-api-access-ddlq8\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.423893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.423733 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-data\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.423893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.423750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e0bf24-3c2f-4161-a254-8c889f0630aa-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.423893 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.423774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-envoy\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424059 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.423988 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-podinfo\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424122 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-token\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424180 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424133 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-certs\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424180 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-socket\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424180 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-credential-socket\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424331 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424190 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-data\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424468 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424445 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-credential-socket\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424549 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e0bf24-3c2f-4161-a254-8c889f0630aa-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-certs\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.424730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.424590 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-socket\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.426110 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.426092 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-envoy\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.426495 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.426477 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-podinfo\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.437295 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.437268 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-token\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.437470 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.437451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlq8\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-kube-api-access-ddlq8\") pod \"e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.532785 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.532763 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:56.808351 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.808327 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79"] Sep 08 18:56:56.811858 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:56:56.811832 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e0bf24_3c2f_4161_a254_8c889f0630aa.slice/crio-6a45565d4afb8dab3ddfece51365fd57d073ed0e87cc944f4e8d5d4fe1607c59 WatchSource:0}: Error finding container 6a45565d4afb8dab3ddfece51365fd57d073ed0e87cc944f4e8d5d4fe1607c59: Status 404 returned error can't find the container with id 6a45565d4afb8dab3ddfece51365fd57d073ed0e87cc944f4e8d5d4fe1607c59 Sep 08 18:56:56.814098 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.814063 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:56:56.814216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.814136 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:56:56.814216 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:56.814162 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:56:57.150851 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:57.150729 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" event={"ID":"41e0bf24-3c2f-4161-a254-8c889f0630aa","Type":"ContainerStarted","Data":"f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e"} Sep 08 18:56:57.150851 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:57.150796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" event={"ID":"41e0bf24-3c2f-4161-a254-8c889f0630aa","Type":"ContainerStarted","Data":"6a45565d4afb8dab3ddfece51365fd57d073ed0e87cc944f4e8d5d4fe1607c59"} Sep 08 18:56:57.172027 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:57.171982 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podStartSLOduration=1.171968951 podStartE2EDuration="1.171968951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:57.170622563 +0000 UTC m=+1375.679774995" watchObservedRunningTime="2026-09-08 18:56:57.171968951 +0000 UTC m=+1375.681121435" Sep 08 18:56:57.533499 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:57.533468 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:57.538582 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:57.538559 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:58.156250 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:58.156215 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:58.157254 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:58.157233 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:56:58.784397 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:58.784354 2557 patch_prober.go:28] interesting pod/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 18:56:58.784833 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:56:58.784414 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 18:57:00.555821 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.555798 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:57:00.662984 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.662906 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-token\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.662984 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.662946 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-podinfo\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.662984 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.662971 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-envoy\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.663243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663002 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-data\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.663243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663063 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-socket\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.663243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663128 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istiod-ca-cert\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.663243 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663160 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-certs\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.663449 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663255 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:00.663449 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663360 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:00.663556 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663467 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:57:00.663612 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663517 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:00.663695 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663608 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwlqh\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-kube-api-access-nwlqh\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.663756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.663707 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-credential-socket\") pod \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\" (UID: \"6ea58ca1-8151-4c6a-9aae-f87e5a63df08\") " Sep 08 18:57:00.664055 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.664032 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:00.664127 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.664090 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.664127 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.664109 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.664228 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.664128 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.664228 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.664142 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.665306 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.665276 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:57:00.665306 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.665284 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:00.665462 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.665312 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:57:00.666108 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.666092 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-kube-api-access-nwlqh" (OuterVolumeSpecName: "kube-api-access-nwlqh") pod "6ea58ca1-8151-4c6a-9aae-f87e5a63df08" (UID: "6ea58ca1-8151-4c6a-9aae-f87e5a63df08"). InnerVolumeSpecName "kube-api-access-nwlqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:57:00.765620 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.765594 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlqh\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-kube-api-access-nwlqh\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.765620 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.765621 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.765805 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.765631 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.765805 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.765640 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:00.765805 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:00.765659 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea58ca1-8151-4c6a-9aae-f87e5a63df08-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:57:01.172568 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.172533 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerID="773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db" exitCode=0 Sep 08 18:57:01.172730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.172574 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" Sep 08 18:57:01.172730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.172615 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" event={"ID":"6ea58ca1-8151-4c6a-9aae-f87e5a63df08","Type":"ContainerDied","Data":"773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db"} Sep 08 18:57:01.172730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.172688 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7" event={"ID":"6ea58ca1-8151-4c6a-9aae-f87e5a63df08","Type":"ContainerDied","Data":"3dc1f167cd450bc91666b3ce6fa65bf2a173ca23d45511471dd607e0a8bd4c52"} Sep 08 18:57:01.172730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.172716 2557 scope.go:117] "RemoveContainer" containerID="773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db" Sep 08 18:57:01.182553 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.182536 2557 scope.go:117] "RemoveContainer" containerID="f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45" Sep 08 18:57:01.190586 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.190570 2557 scope.go:117] "RemoveContainer" containerID="773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db" Sep 08 18:57:01.190866 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:57:01.190850 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db\": container with ID starting with 773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db not found: ID does not exist" containerID="773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db" Sep 08 18:57:01.190934 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.190877 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db"} err="failed to get container status \"773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db\": rpc error: code = NotFound desc = could not find container \"773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db\": container with ID starting with 773de86de792642dba34fda959de62fcfa357c2057f46df8927a0805f30826db not found: ID does not exist" Sep 08 18:57:01.190934 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.190899 2557 scope.go:117] "RemoveContainer" containerID="f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45" Sep 08 18:57:01.191145 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:57:01.191127 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45\": container with ID starting with f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45 not found: ID does not exist" containerID="f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45" Sep 08 18:57:01.191195 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.191152 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45"} err="failed to get container status \"f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45\": rpc error: code = NotFound desc = could not find container \"f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45\": container with ID starting with f41fae04dae1b8094b9ed43a62f3940f8095e8aacaad6453082b314153553d45 not found: ID does not exist" Sep 08 18:57:01.194047 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.194025 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7"] Sep 08 18:57:01.199940 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:01.199920 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3074-openshift-default-6ff985b7fb-zlcl7"] Sep 08 18:57:02.027259 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:02.027220 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" path="/var/lib/kubelet/pods/6ea58ca1-8151-4c6a-9aae-f87e5a63df08/volumes" Sep 08 18:57:13.157581 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.157516 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:57:13.158054 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.157573 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 08 18:57:13.251868 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.251838 2557 generic.go:356] "Generic (PLEG): container finished" podID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerID="f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e" exitCode=0 Sep 08 18:57:13.251993 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.251906 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" event={"ID":"41e0bf24-3c2f-4161-a254-8c889f0630aa","Type":"ContainerDied","Data":"f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e"} Sep 08 18:57:13.252278 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.252264 2557 scope.go:117] "RemoveContainer" containerID="f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e" Sep 08 18:57:13.253265 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.253239 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:13.253360 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.253318 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:13.253421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:13.253365 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:14.260372 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:14.260320 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" event={"ID":"41e0bf24-3c2f-4161-a254-8c889f0630aa","Type":"ContainerStarted","Data":"3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55"} Sep 08 18:57:14.533477 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:14.533379 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:57:14.533816 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:14.533786 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 08 18:57:14.534203 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:14.533716 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:57:15.533704 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:15.533664 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:57:15.534186 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:15.533747 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 08 18:57:16.184269 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184238 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd"] Sep 08 18:57:16.184671 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184639 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" Sep 08 18:57:16.184820 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184696 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" Sep 08 18:57:16.184820 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184760 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" Sep 08 18:57:16.184820 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184769 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" Sep 08 18:57:16.184990 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184894 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" Sep 08 18:57:16.184990 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.184907 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea58ca1-8151-4c6a-9aae-f87e5a63df08" containerName="istio-proxy" Sep 08 18:57:16.188315 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.188289 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.190724 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.190697 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-093570-openshift-default-dockercfg-ptb9r\"" Sep 08 18:57:16.199634 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.199609 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd"] Sep 08 18:57:16.207779 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.207757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-data\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.207881 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.207791 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-credential-socket\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.207881 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.207813 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13616b0-a2bc-4578-824b-65a4114a96c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.207997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.207924 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf65g\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-kube-api-access-nf65g\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.207997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.207968 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-token\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.208108 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.208029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-envoy\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.208108 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.208062 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-podinfo\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.208217 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.208112 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-socket\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.208217 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.208145 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-certs\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309078 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-credential-socket\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309078 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13616b0-a2bc-4578-824b-65a4114a96c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309115 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf65g\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-kube-api-access-nf65g\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-token\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-envoy\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-podinfo\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-socket\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-certs\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309326 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-data\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309724 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309561 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-credential-socket\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309724 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309635 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-data\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309840 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-socket\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.309840 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.309780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-certs\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.310295 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.310268 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13616b0-a2bc-4578-824b-65a4114a96c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.311406 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.311384 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-envoy\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.311743 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.311725 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-podinfo\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.327866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.327823 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-token\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.331596 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.331545 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf65g\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-kube-api-access-nf65g\") pod \"e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.501898 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.501868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:16.545485 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.545447 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:57:16.545904 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.545521 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 08 18:57:16.808717 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.808690 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd"] Sep 08 18:57:16.810252 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:57:16.810230 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13616b0_a2bc_4578_824b_65a4114a96c7.slice/crio-12dcff5dc255319453b0b4c7011b82bac49836b4d755d7e33782eb8059c5548f WatchSource:0}: Error finding container 12dcff5dc255319453b0b4c7011b82bac49836b4d755d7e33782eb8059c5548f: Status 404 returned error can't find the container with id 12dcff5dc255319453b0b4c7011b82bac49836b4d755d7e33782eb8059c5548f Sep 08 18:57:16.812433 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.812406 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:16.812492 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.812467 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:16.812529 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:16.812492 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:17.277175 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.277137 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" event={"ID":"c13616b0-a2bc-4578-824b-65a4114a96c7","Type":"ContainerStarted","Data":"8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5"} Sep 08 18:57:17.277368 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.277191 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" event={"ID":"c13616b0-a2bc-4578-824b-65a4114a96c7","Type":"ContainerStarted","Data":"12dcff5dc255319453b0b4c7011b82bac49836b4d755d7e33782eb8059c5548f"} Sep 08 18:57:17.295400 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.295350 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podStartSLOduration=1.29533088 podStartE2EDuration="1.29533088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:17.294832785 +0000 UTC m=+1395.803985227" watchObservedRunningTime="2026-09-08 18:57:17.29533088 +0000 UTC m=+1395.804483322" Sep 08 18:57:17.502457 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.502423 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:17.507312 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.507287 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:17.533497 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.533417 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:57:17.533738 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:17.533709 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 08 18:57:18.282504 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:18.282465 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:18.283460 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:18.283442 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:18.533797 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:18.533710 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:57:18.534071 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:18.534039 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 08 18:57:19.537363 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:19.537334 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:57:19.537737 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:19.537605 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:57:19.538222 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:19.538203 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 18:57:20.306450 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.306406 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz"] Sep 08 18:57:20.310631 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.310608 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.312760 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.312738 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-093570-serving-cert\"" Sep 08 18:57:20.321273 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.321251 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz"] Sep 08 18:57:20.358867 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.358837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.359011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.358959 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.359011 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.358990 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.359129 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.359098 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qxs\" (UniqueName: \"kubernetes.io/projected/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-kube-api-access-b5qxs\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.460123 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.460090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.460303 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.460138 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.460303 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.460156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.460303 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:57:20.460256 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-serving-cert: secret "maas-api-e2e-shared-a-w2-093570-serving-cert" not found Sep 08 18:57:20.460464 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:57:20.460337 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls podName:1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e nodeName:}" failed. No retries permitted until 2026-09-08 18:57:20.96031597 +0000 UTC m=+1399.469468395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls") pod "maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" (UID: "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e") : secret "maas-api-e2e-shared-a-w2-093570-serving-cert" not found Sep 08 18:57:20.460464 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.460362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qxs\" (UniqueName: \"kubernetes.io/projected/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-kube-api-access-b5qxs\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.460833 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.460815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.462729 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.462700 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.470861 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.470837 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qxs\" (UniqueName: \"kubernetes.io/projected/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-kube-api-access-b5qxs\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.965255 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.965224 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:20.967530 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:20.967506 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:21.226168 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:21.226136 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:21.498793 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:21.498758 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz"] Sep 08 18:57:21.500838 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:57:21.500810 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0fe8ea_54a7_4ab5_af9a_3f20a70e875e.slice/crio-2c241bd5ea7afdf6934125335c2efbee94013941d323af97f1ccc1b93b474891 WatchSource:0}: Error finding container 2c241bd5ea7afdf6934125335c2efbee94013941d323af97f1ccc1b93b474891: Status 404 returned error can't find the container with id 2c241bd5ea7afdf6934125335c2efbee94013941d323af97f1ccc1b93b474891 Sep 08 18:57:21.502204 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:21.502188 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:57:22.307095 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:22.307018 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" event={"ID":"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e","Type":"ContainerStarted","Data":"5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e"} Sep 08 18:57:22.307428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:22.307113 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" event={"ID":"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e","Type":"ContainerStarted","Data":"2c241bd5ea7afdf6934125335c2efbee94013941d323af97f1ccc1b93b474891"} Sep 08 18:57:22.307428 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:22.307139 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:24.806520 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.806466 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" podStartSLOduration=4.273197797 podStartE2EDuration="4.806449216s" totalImagesPullingTime="533.251419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:22.3233761 +0000 UTC m=+1400.832528541" watchObservedRunningTime="2026-09-08 18:57:24.806449216 +0000 UTC m=+1403.315601656" Sep 08 18:57:24.807574 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.807545 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg"] Sep 08 18:57:24.811323 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.811303 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.813707 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.813684 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-525196-openshift-default-dockercfg-xltf4\"" Sep 08 18:57:24.823276 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.822368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg"] Sep 08 18:57:24.900478 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zszx\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-kube-api-access-8zszx\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900641 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900504 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-credential-socket\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900641 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85dd9520-c83c-4783-81e0-b2142fba8d83-istiod-ca-cert\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900636 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-certs\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900692 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-podinfo\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900814 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900745 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-token\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-socket\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900901 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-data\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:24.900972 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:24.900953 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-envoy\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.001768 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.001740 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zszx\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-kube-api-access-8zszx\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.001945 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.001776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-credential-socket\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.001945 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.001794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85dd9520-c83c-4783-81e0-b2142fba8d83-istiod-ca-cert\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.001945 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.001929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-certs\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002148 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.001970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-podinfo\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002148 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.001991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-token\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002148 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002046 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-socket\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002148 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-data\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002148 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002113 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-envoy\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002374 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002213 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-credential-socket\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002425 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002402 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-socket\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002465 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002427 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-certs\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002508 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002482 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-data\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.002598 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.002579 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85dd9520-c83c-4783-81e0-b2142fba8d83-istiod-ca-cert\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.004292 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.004264 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-envoy\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.004483 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.004466 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-podinfo\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.009567 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.009546 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zszx\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-kube-api-access-8zszx\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.009831 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.009811 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-token\") pod \"e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.124629 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.124541 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:25.406601 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.406574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg"] Sep 08 18:57:25.409629 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:57:25.409604 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dd9520_c83c_4783_81e0_b2142fba8d83.slice/crio-943a3f434bc433a5b64aca5ae2357bf2cccbe7d4101cf9cad492eb93f5c6e99a WatchSource:0}: Error finding container 943a3f434bc433a5b64aca5ae2357bf2cccbe7d4101cf9cad492eb93f5c6e99a: Status 404 returned error can't find the container with id 943a3f434bc433a5b64aca5ae2357bf2cccbe7d4101cf9cad492eb93f5c6e99a Sep 08 18:57:25.412061 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.412030 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:25.412145 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.412108 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:25.412202 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:25.412153 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:26.330985 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:26.330942 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" event={"ID":"85dd9520-c83c-4783-81e0-b2142fba8d83","Type":"ContainerStarted","Data":"161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25"} Sep 08 18:57:26.331442 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:26.330996 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" event={"ID":"85dd9520-c83c-4783-81e0-b2142fba8d83","Type":"ContainerStarted","Data":"943a3f434bc433a5b64aca5ae2357bf2cccbe7d4101cf9cad492eb93f5c6e99a"} Sep 08 18:57:26.355258 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:26.355209 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podStartSLOduration=2.355196488 podStartE2EDuration="2.355196488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:26.353218602 +0000 UTC m=+1404.862371043" watchObservedRunningTime="2026-09-08 18:57:26.355196488 +0000 UTC m=+1404.864348929" Sep 08 18:57:27.126936 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:27.126900 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:27.141272 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:27.141242 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:27.339668 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:27.339615 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:27.344371 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:27.344343 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:28.324910 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:28.324881 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 18:57:34.323810 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:34.323770 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 18:57:34.324197 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:34.323833 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 18:57:35.384560 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:35.384508 2557 generic.go:356] "Generic (PLEG): container finished" podID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerID="8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5" exitCode=0 Sep 08 18:57:35.384898 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:35.384580 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" event={"ID":"c13616b0-a2bc-4578-824b-65a4114a96c7","Type":"ContainerDied","Data":"8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5"} Sep 08 18:57:35.384988 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:35.384974 2557 scope.go:117] "RemoveContainer" containerID="8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5" Sep 08 18:57:35.385991 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:35.385957 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:35.386086 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:35.386044 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:35.386149 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:35.386091 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:36.391696 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:36.391662 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" event={"ID":"c13616b0-a2bc-4578-824b-65a4114a96c7","Type":"ContainerStarted","Data":"856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948"} Sep 08 18:57:36.502072 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:36.502037 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:36.502286 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:36.502263 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 18:57:36.502336 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:36.502315 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 18:57:37.502930 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:37.502888 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 18:57:37.503330 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:37.502950 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 18:57:38.502855 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:38.502819 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 18:57:38.503099 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:38.503071 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 18:57:39.502404 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:39.502364 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 18:57:39.502578 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:39.502421 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 18:57:40.502979 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:40.502941 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 18:57:40.503415 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:40.503000 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 18:57:41.502456 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:41.502420 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 18:57:41.502620 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:41.502474 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 18:57:42.505730 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:42.505700 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:42.506112 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:42.506025 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:42.506674 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:42.506638 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 18:57:46.459999 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:46.459919 2557 generic.go:356] "Generic (PLEG): container finished" podID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerID="161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25" exitCode=0 Sep 08 18:57:46.460379 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:46.459993 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" event={"ID":"85dd9520-c83c-4783-81e0-b2142fba8d83","Type":"ContainerDied","Data":"161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25"} Sep 08 18:57:46.460379 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:46.460338 2557 scope.go:117] "RemoveContainer" containerID="161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25" Sep 08 18:57:46.461377 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:46.461351 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:46.461453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:46.461419 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:46.461453 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:46.461448 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:57:47.469033 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:47.468998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" event={"ID":"85dd9520-c83c-4783-81e0-b2142fba8d83","Type":"ContainerStarted","Data":"7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55"} Sep 08 18:57:48.125791 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:48.125754 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:48.125993 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:48.125869 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 18:57:48.125993 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:48.125913 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 18:57:49.125503 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:49.125466 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 18:57:49.126002 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:49.125539 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 18:57:50.128344 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:50.128308 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 18:57:50.128919 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:50.128885 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 18:57:51.125190 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:51.125153 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 18:57:51.125472 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:51.125445 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 18:57:52.125235 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:52.125196 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 18:57:52.125728 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:52.125263 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 18:57:53.129073 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:53.129044 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:53.129543 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:53.129525 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:57:53.130135 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:57:53.130113 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 18:58:14.344716 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.344685 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-7e7b52"] Sep 08 18:58:14.348225 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.348206 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:14.350675 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.350628 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\"" Sep 08 18:58:14.352607 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.352584 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-7e7b52"] Sep 08 18:58:14.460492 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.460468 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgg7c\" (UniqueName: \"kubernetes.io/projected/f3cdf638-36dd-4f13-839b-b561b2ff6b6e-kube-api-access-xgg7c\") pod \"test-curl-7692-7e7b52\" (UID: \"f3cdf638-36dd-4f13-839b-b561b2ff6b6e\") " pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:14.561050 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.561026 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgg7c\" (UniqueName: \"kubernetes.io/projected/f3cdf638-36dd-4f13-839b-b561b2ff6b6e-kube-api-access-xgg7c\") pod \"test-curl-7692-7e7b52\" (UID: \"f3cdf638-36dd-4f13-839b-b561b2ff6b6e\") " pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:14.569592 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.569563 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgg7c\" (UniqueName: \"kubernetes.io/projected/f3cdf638-36dd-4f13-839b-b561b2ff6b6e-kube-api-access-xgg7c\") pod \"test-curl-7692-7e7b52\" (UID: \"f3cdf638-36dd-4f13-839b-b561b2ff6b6e\") " pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:14.659907 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.659843 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:14.927459 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:14.927382 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-7e7b52"] Sep 08 18:58:14.930582 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:58:14.930558 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3cdf638_36dd_4f13_839b_b561b2ff6b6e.slice/crio-dad55b4a3658f2b9bf93c37b512b55063bd016888f463f7e47efa35c4fec4449 WatchSource:0}: Error finding container dad55b4a3658f2b9bf93c37b512b55063bd016888f463f7e47efa35c4fec4449: Status 404 returned error can't find the container with id dad55b4a3658f2b9bf93c37b512b55063bd016888f463f7e47efa35c4fec4449 Sep 08 18:58:15.630018 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:15.629986 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-7e7b52" event={"ID":"f3cdf638-36dd-4f13-839b-b561b2ff6b6e","Type":"ContainerStarted","Data":"93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f"} Sep 08 18:58:15.630380 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:15.630029 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-7e7b52" event={"ID":"f3cdf638-36dd-4f13-839b-b561b2ff6b6e","Type":"ContainerStarted","Data":"dad55b4a3658f2b9bf93c37b512b55063bd016888f463f7e47efa35c4fec4449"} Sep 08 18:58:15.643690 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:15.643624 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-7e7b52" podStartSLOduration=1.643608767 podStartE2EDuration="1.643608767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:15.643003741 +0000 UTC m=+1454.152156178" watchObservedRunningTime="2026-09-08 18:58:15.643608767 +0000 UTC m=+1454.152761209" Sep 08 18:58:16.264421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.264378 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-7e7b52"] Sep 08 18:58:16.266790 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.266760 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-7e7b52"] Sep 08 18:58:16.395620 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.395559 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-e69a5b"] Sep 08 18:58:16.395968 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.395955 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" containerName="test-curl-7692-7e7b52" Sep 08 18:58:16.396020 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.395970 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" containerName="test-curl-7692-7e7b52" Sep 08 18:58:16.396055 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.396047 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" containerName="test-curl-7692-7e7b52" Sep 08 18:58:16.399111 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.399096 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:16.404530 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.404505 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-e69a5b"] Sep 08 18:58:16.478004 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.477967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlqf\" (UniqueName: \"kubernetes.io/projected/06860875-dc00-4f74-8867-9ae2174356e1-kube-api-access-zmlqf\") pod \"test-curl-7692-e69a5b\" (UID: \"06860875-dc00-4f74-8867-9ae2174356e1\") " pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:16.578640 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.578618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlqf\" (UniqueName: \"kubernetes.io/projected/06860875-dc00-4f74-8867-9ae2174356e1-kube-api-access-zmlqf\") pod \"test-curl-7692-e69a5b\" (UID: \"06860875-dc00-4f74-8867-9ae2174356e1\") " pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:16.586277 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.586256 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlqf\" (UniqueName: \"kubernetes.io/projected/06860875-dc00-4f74-8867-9ae2174356e1-kube-api-access-zmlqf\") pod \"test-curl-7692-e69a5b\" (UID: \"06860875-dc00-4f74-8867-9ae2174356e1\") " pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:16.635877 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:16.635847 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-7e7b52" container="test-curl-7692-7e7b52" Sep 08 18:58:16.709191 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.709135 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:16.977399 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:16.977326 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-e69a5b"] Sep 08 18:58:16.979602 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:58:16.979575 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06860875_dc00_4f74_8867_9ae2174356e1.slice/crio-397ef2698690e54fee232c5e658d6fb3962926a1b821611c3e1b0a82ac2aa3e5 WatchSource:0}: Error finding container 397ef2698690e54fee232c5e658d6fb3962926a1b821611c3e1b0a82ac2aa3e5: Status 404 returned error can't find the container with id 397ef2698690e54fee232c5e658d6fb3962926a1b821611c3e1b0a82ac2aa3e5 Sep 08 18:58:17.642749 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:17.642711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-e69a5b" event={"ID":"06860875-dc00-4f74-8867-9ae2174356e1","Type":"ContainerStarted","Data":"7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b"} Sep 08 18:58:17.643122 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:17.642761 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-e69a5b" event={"ID":"06860875-dc00-4f74-8867-9ae2174356e1","Type":"ContainerStarted","Data":"397ef2698690e54fee232c5e658d6fb3962926a1b821611c3e1b0a82ac2aa3e5"} Sep 08 18:58:17.643122 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:17.642840 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-7e7b52" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" containerName="test-curl-7692-7e7b52" containerID="cri-o://93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f" gracePeriod=2 Sep 08 18:58:17.645012 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:17.644987 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-7e7b52\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:17.659227 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:17.659185 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-e69a5b" podStartSLOduration=1.659168837 podStartE2EDuration="1.659168837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:17.658180895 +0000 UTC m=+1456.167333337" watchObservedRunningTime="2026-09-08 18:58:17.659168837 +0000 UTC m=+1456.168321277" Sep 08 18:58:17.659961 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:17.659936 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-7e7b52\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:18.305866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:18.305835 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-e69a5b"] Sep 08 18:58:18.313080 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:18.313000 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-e69a5b"] Sep 08 18:58:18.647863 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:18.647787 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-e69a5b" container="test-curl-7692-e69a5b" Sep 08 18:58:19.654406 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:19.654365 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-e69a5b" podUID="06860875-dc00-4f74-8867-9ae2174356e1" containerName="test-curl-7692-e69a5b" containerID="cri-o://7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b" gracePeriod=2 Sep 08 18:58:19.660542 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:19.660513 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-e69a5b\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="06860875-dc00-4f74-8867-9ae2174356e1" pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:19.785007 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:19.784986 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:19.786904 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:19.786879 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-e69a5b\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="06860875-dc00-4f74-8867-9ae2174356e1" pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:19.906616 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:19.906555 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgg7c\" (UniqueName: \"kubernetes.io/projected/f3cdf638-36dd-4f13-839b-b561b2ff6b6e-kube-api-access-xgg7c\") pod \"f3cdf638-36dd-4f13-839b-b561b2ff6b6e\" (UID: \"f3cdf638-36dd-4f13-839b-b561b2ff6b6e\") " Sep 08 18:58:19.908435 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:19.908414 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cdf638-36dd-4f13-839b-b561b2ff6b6e-kube-api-access-xgg7c" (OuterVolumeSpecName: "kube-api-access-xgg7c") pod "f3cdf638-36dd-4f13-839b-b561b2ff6b6e" (UID: "f3cdf638-36dd-4f13-839b-b561b2ff6b6e"). InnerVolumeSpecName "kube-api-access-xgg7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:58:20.007609 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.007585 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgg7c\" (UniqueName: \"kubernetes.io/projected/f3cdf638-36dd-4f13-839b-b561b2ff6b6e-kube-api-access-xgg7c\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:58:20.026001 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.025971 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" path="/var/lib/kubelet/pods/f3cdf638-36dd-4f13-839b-b561b2ff6b6e/volumes" Sep 08 18:58:20.194339 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.194287 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-39e833"] Sep 08 18:58:20.194712 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.194692 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06860875-dc00-4f74-8867-9ae2174356e1" containerName="test-curl-7692-e69a5b" Sep 08 18:58:20.194712 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.194713 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="06860875-dc00-4f74-8867-9ae2174356e1" containerName="test-curl-7692-e69a5b" Sep 08 18:58:20.194894 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.194852 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="06860875-dc00-4f74-8867-9ae2174356e1" containerName="test-curl-7692-e69a5b" Sep 08 18:58:20.197965 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.197942 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:20.201836 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.201807 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-39e833"] Sep 08 18:58:20.309429 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.309406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvfn4\" (UniqueName: \"kubernetes.io/projected/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee-kube-api-access-lvfn4\") pod \"test-curl-7692-39e833\" (UID: \"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee\") " pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:20.410553 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.410527 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvfn4\" (UniqueName: \"kubernetes.io/projected/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee-kube-api-access-lvfn4\") pod \"test-curl-7692-39e833\" (UID: \"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee\") " pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:20.419039 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.419019 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvfn4\" (UniqueName: \"kubernetes.io/projected/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee-kube-api-access-lvfn4\") pod \"test-curl-7692-39e833\" (UID: \"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee\") " pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:20.509156 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.509135 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:20.660686 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.660629 2557 generic.go:356] "Generic (PLEG): container finished" podID="f3cdf638-36dd-4f13-839b-b561b2ff6b6e" containerID="93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f" exitCode=137 Sep 08 18:58:20.661040 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.660693 2557 scope.go:117] "RemoveContainer" containerID="93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f" Sep 08 18:58:20.661040 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.660692 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7e7b52" Sep 08 18:58:20.668857 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.668842 2557 scope.go:117] "RemoveContainer" containerID="93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f" Sep 08 18:58:20.669091 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:20.669074 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f\": container with ID starting with 93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f not found: ID does not exist" containerID="93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f" Sep 08 18:58:20.669147 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.669098 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f"} err="failed to get container status \"93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f\": rpc error: code = NotFound desc = could not find container \"93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f\": container with ID starting with 93cee9fb2b22e31bcb24787ab4537df2cb58d19d821c009eb06d35c4e985d70f not found: ID does not exist" Sep 08 18:58:20.777911 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:20.777893 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-39e833"] Sep 08 18:58:20.780060 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:58:20.780042 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3ea01c9_5ce7_4f0a_9ebc_d72a139e3bee.slice/crio-9c2bce4af042109c216e79392ec20745f70f3af6813208d5ca0bf2e1fd06ced9 WatchSource:0}: Error finding container 9c2bce4af042109c216e79392ec20745f70f3af6813208d5ca0bf2e1fd06ced9: Status 404 returned error can't find the container with id 9c2bce4af042109c216e79392ec20745f70f3af6813208d5ca0bf2e1fd06ced9 Sep 08 18:58:21.667637 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:21.667607 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-39e833" event={"ID":"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee","Type":"ContainerStarted","Data":"0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4"} Sep 08 18:58:21.667989 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:21.667681 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-39e833" event={"ID":"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee","Type":"ContainerStarted","Data":"9c2bce4af042109c216e79392ec20745f70f3af6813208d5ca0bf2e1fd06ced9"} Sep 08 18:58:21.685317 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:21.685281 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-39e833" podStartSLOduration=1.6852658090000001 podStartE2EDuration="1.685265809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:21.683011689 +0000 UTC m=+1460.192164129" watchObservedRunningTime="2026-09-08 18:58:21.685265809 +0000 UTC m=+1460.194418250" Sep 08 18:58:21.844967 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:21.844946 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:21.921714 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:21.921634 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmlqf\" (UniqueName: \"kubernetes.io/projected/06860875-dc00-4f74-8867-9ae2174356e1-kube-api-access-zmlqf\") pod \"06860875-dc00-4f74-8867-9ae2174356e1\" (UID: \"06860875-dc00-4f74-8867-9ae2174356e1\") " Sep 08 18:58:21.923384 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:21.923359 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06860875-dc00-4f74-8867-9ae2174356e1-kube-api-access-zmlqf" (OuterVolumeSpecName: "kube-api-access-zmlqf") pod "06860875-dc00-4f74-8867-9ae2174356e1" (UID: "06860875-dc00-4f74-8867-9ae2174356e1"). InnerVolumeSpecName "kube-api-access-zmlqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:58:22.022981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.022958 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmlqf\" (UniqueName: \"kubernetes.io/projected/06860875-dc00-4f74-8867-9ae2174356e1-kube-api-access-zmlqf\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:58:22.025617 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.025595 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06860875-dc00-4f74-8867-9ae2174356e1" path="/var/lib/kubelet/pods/06860875-dc00-4f74-8867-9ae2174356e1/volumes" Sep 08 18:58:22.281715 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.281635 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-39e833"] Sep 08 18:58:22.284437 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.284363 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-39e833"] Sep 08 18:58:22.675488 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.675407 2557 generic.go:356] "Generic (PLEG): container finished" podID="06860875-dc00-4f74-8867-9ae2174356e1" containerID="7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b" exitCode=137 Sep 08 18:58:22.675488 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.675472 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-e69a5b" Sep 08 18:58:22.675938 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.675511 2557 scope.go:117] "RemoveContainer" containerID="7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b" Sep 08 18:58:22.675938 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:22.675631 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-39e833" container="test-curl-7692-39e833" Sep 08 18:58:22.684051 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.684031 2557 scope.go:117] "RemoveContainer" containerID="7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b" Sep 08 18:58:22.684354 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:22.684331 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b\": container with ID starting with 7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b not found: ID does not exist" containerID="7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b" Sep 08 18:58:22.684407 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:22.684362 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b"} err="failed to get container status \"7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b\": rpc error: code = NotFound desc = could not find container \"7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b\": container with ID starting with 7e338978f06dc545c225bd652093f938b5b4dfbbca5ba5a32450e4780678248b not found: ID does not exist" Sep 08 18:58:23.264690 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.264635 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl"] Sep 08 18:58:23.265341 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.265314 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" containerName="test-curl-7692-39e833" Sep 08 18:58:23.265341 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.265344 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" containerName="test-curl-7692-39e833" Sep 08 18:58:23.265522 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.265474 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" containerName="test-curl-7692-39e833" Sep 08 18:58:23.270372 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.270347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.273275 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.273250 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3adca991-openshift-default-dockercfg-ttqck\"" Sep 08 18:58:23.279745 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.279724 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl"] Sep 08 18:58:23.332716 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-socket\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.332866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332754 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-certs\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.332866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqfw\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-kube-api-access-wgqfw\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.332866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332859 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bce6761-613e-49c9-9f87-d989d199c653-istio-podinfo\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.332993 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bce6761-613e-49c9-9f87-d989d199c653-istiod-ca-cert\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.332993 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332915 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-credential-socket\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.332993 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-envoy\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.333096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.332991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-data\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.333096 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.333069 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-istio-token\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434565 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434527 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-envoy\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434772 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434579 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-data\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434772 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434635 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-istio-token\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434772 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-socket\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434772 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434755 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-certs\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434785 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqfw\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-kube-api-access-wgqfw\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bce6761-613e-49c9-9f87-d989d199c653-istio-podinfo\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bce6761-613e-49c9-9f87-d989d199c653-istiod-ca-cert\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.434981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.434901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-credential-socket\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.435208 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.435009 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-data\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.435493 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.435464 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-socket\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.435604 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.435576 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-certs\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.435776 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.435750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bce6761-613e-49c9-9f87-d989d199c653-istiod-ca-cert\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.435860 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.435750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-credential-socket\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.437115 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.437094 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-envoy\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.437336 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.437314 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bce6761-613e-49c9-9f87-d989d199c653-istio-podinfo\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.442349 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.442330 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-istio-token\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.442422 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.442364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqfw\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-kube-api-access-wgqfw\") pod \"e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.581592 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.581524 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:23.681854 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.681792 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-39e833" podUID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" containerName="test-curl-7692-39e833" containerID="cri-o://0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4" gracePeriod=2 Sep 08 18:58:23.684166 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:23.684137 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-39e833\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:23.865987 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.865952 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl"] Sep 08 18:58:23.868392 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:58:23.868365 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bce6761_613e_49c9_9f87_d989d199c653.slice/crio-09ac6c906f04a3db6a9aef30935204e939b735e89c537db3c51c8d23746cbd9c WatchSource:0}: Error finding container 09ac6c906f04a3db6a9aef30935204e939b735e89c537db3c51c8d23746cbd9c: Status 404 returned error can't find the container with id 09ac6c906f04a3db6a9aef30935204e939b735e89c537db3c51c8d23746cbd9c Sep 08 18:58:23.870997 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.870958 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:58:23.871078 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.871041 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:58:23.871117 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:23.871086 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:58:24.688911 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:24.688882 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" event={"ID":"3bce6761-613e-49c9-9f87-d989d199c653","Type":"ContainerStarted","Data":"621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4"} Sep 08 18:58:24.689355 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:24.688937 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" event={"ID":"3bce6761-613e-49c9-9f87-d989d199c653","Type":"ContainerStarted","Data":"09ac6c906f04a3db6a9aef30935204e939b735e89c537db3c51c8d23746cbd9c"} Sep 08 18:58:24.708445 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:24.708387 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podStartSLOduration=1.708373411 podStartE2EDuration="1.708373411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:24.707143069 +0000 UTC m=+1463.216295512" watchObservedRunningTime="2026-09-08 18:58:24.708373411 +0000 UTC m=+1463.217525851" Sep 08 18:58:25.582033 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.581998 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:25.587136 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.587112 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:25.694905 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.694878 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:25.695866 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.695846 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:25.838981 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.838918 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:25.956183 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.956151 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvfn4\" (UniqueName: \"kubernetes.io/projected/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee-kube-api-access-lvfn4\") pod \"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee\" (UID: \"f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee\") " Sep 08 18:58:25.958189 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:25.958154 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee-kube-api-access-lvfn4" (OuterVolumeSpecName: "kube-api-access-lvfn4") pod "f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" (UID: "f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee"). InnerVolumeSpecName "kube-api-access-lvfn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:58:26.026959 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.026930 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" path="/var/lib/kubelet/pods/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee/volumes" Sep 08 18:58:26.057194 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.057173 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvfn4\" (UniqueName: \"kubernetes.io/projected/f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee-kube-api-access-lvfn4\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 18:58:26.701284 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.701256 2557 generic.go:356] "Generic (PLEG): container finished" podID="f3ea01c9-5ce7-4f0a-9ebc-d72a139e3bee" containerID="0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4" exitCode=137 Sep 08 18:58:26.701718 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.701295 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-39e833" Sep 08 18:58:26.701718 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.701336 2557 scope.go:117] "RemoveContainer" containerID="0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4" Sep 08 18:58:26.710109 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.710088 2557 scope.go:117] "RemoveContainer" containerID="0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4" Sep 08 18:58:26.710386 ip-10-0-134-251 kubenswrapper[2557]: E0908 18:58:26.710366 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4\": container with ID starting with 0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4 not found: ID does not exist" containerID="0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4" Sep 08 18:58:26.710451 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:26.710395 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4"} err="failed to get container status \"0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4\": rpc error: code = NotFound desc = could not find container \"0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4\": container with ID starting with 0950a3eefb929c62ccbeedafccba81545d6f714e90c25a762073d583b13eb3e4 not found: ID does not exist" Sep 08 18:58:28.030057 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.030026 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9"] Sep 08 18:58:28.034886 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.034868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.039104 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.039085 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3adca991-serving-cert\"" Sep 08 18:58:28.044000 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.043972 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9"] Sep 08 18:58:28.175621 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.175586 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1796e83f-d71c-47c2-acf4-bb22e76379e6-openshift-service-ca\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.175621 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.175617 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q576\" (UniqueName: \"kubernetes.io/projected/1796e83f-d71c-47c2-acf4-bb22e76379e6-kube-api-access-9q576\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.175959 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.175744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-metrics-certs\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.175959 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.175803 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-maas-api-tls\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.277113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.277085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1796e83f-d71c-47c2-acf4-bb22e76379e6-openshift-service-ca\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.277113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.277114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q576\" (UniqueName: \"kubernetes.io/projected/1796e83f-d71c-47c2-acf4-bb22e76379e6-kube-api-access-9q576\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.277297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.277163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-metrics-certs\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.277297 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.277188 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-maas-api-tls\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.277662 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.277630 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1796e83f-d71c-47c2-acf4-bb22e76379e6-openshift-service-ca\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.279641 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.279623 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-maas-api-tls\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.279798 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.279778 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-metrics-certs\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.291585 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.291532 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q576\" (UniqueName: \"kubernetes.io/projected/1796e83f-d71c-47c2-acf4-bb22e76379e6-kube-api-access-9q576\") pod \"maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.346385 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.346357 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:28.631113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.631071 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9"] Sep 08 18:58:28.631680 ip-10-0-134-251 kubenswrapper[2557]: W0908 18:58:28.631639 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1796e83f_d71c_47c2_acf4_bb22e76379e6.slice/crio-e0ad9b1696163b880d28e0760b850129195a368c8c9369f129aa12c653cbd894 WatchSource:0}: Error finding container e0ad9b1696163b880d28e0760b850129195a368c8c9369f129aa12c653cbd894: Status 404 returned error can't find the container with id e0ad9b1696163b880d28e0760b850129195a368c8c9369f129aa12c653cbd894 Sep 08 18:58:28.715599 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:28.715563 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" event={"ID":"1796e83f-d71c-47c2-acf4-bb22e76379e6","Type":"ContainerStarted","Data":"e0ad9b1696163b880d28e0760b850129195a368c8c9369f129aa12c653cbd894"} Sep 08 18:58:29.724327 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:29.724295 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" event={"ID":"1796e83f-d71c-47c2-acf4-bb22e76379e6","Type":"ContainerStarted","Data":"a2197463aeb2cfc8b35d027dfe6e036b2ba455573fef245ef927d562785b855a"} Sep 08 18:58:29.724762 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:29.724388 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:29.746122 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:29.746073 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" podStartSLOduration=1.299918931 podStartE2EDuration="1.746059182s" totalImagesPullingTime="446.140251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:29.743846767 +0000 UTC m=+1468.252999208" watchObservedRunningTime="2026-09-08 18:58:29.746059182 +0000 UTC m=+1468.255211642" Sep 08 18:58:35.735229 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:35.735201 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 18:58:46.832360 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:46.832324 2557 generic.go:356] "Generic (PLEG): container finished" podID="3bce6761-613e-49c9-9f87-d989d199c653" containerID="621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4" exitCode=0 Sep 08 18:58:46.833012 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:46.832397 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" event={"ID":"3bce6761-613e-49c9-9f87-d989d199c653","Type":"ContainerDied","Data":"621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4"} Sep 08 18:58:46.833012 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:46.832790 2557 scope.go:117] "RemoveContainer" containerID="621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4" Sep 08 18:58:46.834113 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:46.834078 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:58:46.834186 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:46.834169 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:58:46.834223 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:46.834211 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 18:58:47.845319 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:47.845273 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" event={"ID":"3bce6761-613e-49c9-9f87-d989d199c653","Type":"ContainerStarted","Data":"3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d"} Sep 08 18:58:48.581942 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:48.581903 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:48.582143 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:48.582097 2557 patch_prober.go:28] interesting pod/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:58:48.582218 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:48.582149 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 08 18:58:49.582421 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:49.582378 2557 patch_prober.go:28] interesting pod/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:58:49.582816 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:49.582452 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 08 18:58:50.582600 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:50.582563 2557 patch_prober.go:28] interesting pod/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:58:50.583001 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:50.582625 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 08 18:58:51.582475 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:51.582437 2557 patch_prober.go:28] interesting pod/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:58:51.582676 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:51.582491 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 08 18:58:52.582756 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:52.582716 2557 patch_prober.go:28] interesting pod/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:58:52.583123 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:52.582777 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 08 18:58:53.585872 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:53.585845 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:53.586227 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:53.586208 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 18:58:53.586932 ip-10-0-134-251 kubenswrapper[2557]: I0908 18:58:53.586914 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 19:00:00.143726 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:00.143695 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw"] Sep 08 19:00:00.147305 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:00.147288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" Sep 08 19:00:00.149402 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:00.149376 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:00:00.153559 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:00.153537 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw"] Sep 08 19:00:00.159860 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:00.159838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" Sep 08 19:00:00.438686 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:00.438661 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw"] Sep 08 19:00:00.440574 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:00:00.440548 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb4b788_e71b_44d5_8f53_8c72d109ebc6.slice/crio-0e704d25f34affd8cf7de8ef5f73d3077941178f382676e44ac2e5a79944a750 WatchSource:0}: Error finding container 0e704d25f34affd8cf7de8ef5f73d3077941178f382676e44ac2e5a79944a750: Status 404 returned error can't find the container with id 0e704d25f34affd8cf7de8ef5f73d3077941178f382676e44ac2e5a79944a750 Sep 08 19:00:01.241007 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:01.240977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" event={"ID":"fbb4b788-e71b-44d5-8f53-8c72d109ebc6","Type":"ContainerStarted","Data":"0e704d25f34affd8cf7de8ef5f73d3077941178f382676e44ac2e5a79944a750"} Sep 08 19:00:02.116379 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.116350 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9"] Sep 08 19:00:02.116606 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.116560 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" podUID="1796e83f-d71c-47c2-acf4-bb22e76379e6" containerName="maas-api" containerID="cri-o://a2197463aeb2cfc8b35d027dfe6e036b2ba455573fef245ef927d562785b855a" gracePeriod=30 Sep 08 19:00:02.150732 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.150706 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw"] Sep 08 19:00:02.250628 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.250599 2557 generic.go:356] "Generic (PLEG): container finished" podID="1796e83f-d71c-47c2-acf4-bb22e76379e6" containerID="a2197463aeb2cfc8b35d027dfe6e036b2ba455573fef245ef927d562785b855a" exitCode=0 Sep 08 19:00:02.251169 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.250676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" event={"ID":"1796e83f-d71c-47c2-acf4-bb22e76379e6","Type":"ContainerDied","Data":"a2197463aeb2cfc8b35d027dfe6e036b2ba455573fef245ef927d562785b855a"} Sep 08 19:00:02.370615 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.370557 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 19:00:02.483266 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.483234 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q576\" (UniqueName: \"kubernetes.io/projected/1796e83f-d71c-47c2-acf4-bb22e76379e6-kube-api-access-9q576\") pod \"1796e83f-d71c-47c2-acf4-bb22e76379e6\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " Sep 08 19:00:02.483431 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.483307 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-maas-api-tls\") pod \"1796e83f-d71c-47c2-acf4-bb22e76379e6\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " Sep 08 19:00:02.483431 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.483348 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-metrics-certs\") pod \"1796e83f-d71c-47c2-acf4-bb22e76379e6\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " Sep 08 19:00:02.483546 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.483424 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1796e83f-d71c-47c2-acf4-bb22e76379e6-openshift-service-ca\") pod \"1796e83f-d71c-47c2-acf4-bb22e76379e6\" (UID: \"1796e83f-d71c-47c2-acf4-bb22e76379e6\") " Sep 08 19:00:02.483812 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.483785 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1796e83f-d71c-47c2-acf4-bb22e76379e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1796e83f-d71c-47c2-acf4-bb22e76379e6" (UID: "1796e83f-d71c-47c2-acf4-bb22e76379e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:00:02.485404 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.485373 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1796e83f-d71c-47c2-acf4-bb22e76379e6" (UID: "1796e83f-d71c-47c2-acf4-bb22e76379e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:00:02.485404 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.485390 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1796e83f-d71c-47c2-acf4-bb22e76379e6" (UID: "1796e83f-d71c-47c2-acf4-bb22e76379e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:00:02.485582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.485382 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1796e83f-d71c-47c2-acf4-bb22e76379e6-kube-api-access-9q576" (OuterVolumeSpecName: "kube-api-access-9q576") pod "1796e83f-d71c-47c2-acf4-bb22e76379e6" (UID: "1796e83f-d71c-47c2-acf4-bb22e76379e6"). InnerVolumeSpecName "kube-api-access-9q576". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:00:02.584820 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.584789 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:00:02.584820 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.584813 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1796e83f-d71c-47c2-acf4-bb22e76379e6-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:00:02.584820 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.584822 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1796e83f-d71c-47c2-acf4-bb22e76379e6-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:00:02.585038 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:02.584831 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q576\" (UniqueName: \"kubernetes.io/projected/1796e83f-d71c-47c2-acf4-bb22e76379e6-kube-api-access-9q576\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:00:03.257248 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:03.257215 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" Sep 08 19:00:03.257692 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:03.257243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9" event={"ID":"1796e83f-d71c-47c2-acf4-bb22e76379e6","Type":"ContainerDied","Data":"e0ad9b1696163b880d28e0760b850129195a368c8c9369f129aa12c653cbd894"} Sep 08 19:00:03.257692 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:03.257305 2557 scope.go:117] "RemoveContainer" containerID="a2197463aeb2cfc8b35d027dfe6e036b2ba455573fef245ef927d562785b855a" Sep 08 19:00:03.278547 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:03.278525 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9"] Sep 08 19:00:03.282775 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:03.282752 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3adca991-d89cf6cbd-kdbp9"] Sep 08 19:00:04.026517 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:04.026483 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1796e83f-d71c-47c2-acf4-bb22e76379e6" path="/var/lib/kubelet/pods/1796e83f-d71c-47c2-acf4-bb22e76379e6/volumes" Sep 08 19:00:18.342110 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:18.342080 2557 generic.go:356] "Generic (PLEG): container finished" podID="fbb4b788-e71b-44d5-8f53-8c72d109ebc6" containerID="6136336a10fe757a000770110a8287fa7fd9fae6d41c26183259ed8b746000fb" exitCode=6 Sep 08 19:00:18.342438 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:18.342165 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" event={"ID":"fbb4b788-e71b-44d5-8f53-8c72d109ebc6","Type":"ContainerDied","Data":"6136336a10fe757a000770110a8287fa7fd9fae6d41c26183259ed8b746000fb"} Sep 08 19:00:18.483426 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:18.483399 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" Sep 08 19:00:19.348617 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:19.348588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" event={"ID":"fbb4b788-e71b-44d5-8f53-8c72d109ebc6","Type":"ContainerDied","Data":"0e704d25f34affd8cf7de8ef5f73d3077941178f382676e44ac2e5a79944a750"} Sep 08 19:00:19.348617 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:19.348610 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw" Sep 08 19:00:19.349087 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:19.348642 2557 scope.go:117] "RemoveContainer" containerID="6136336a10fe757a000770110a8287fa7fd9fae6d41c26183259ed8b746000fb" Sep 08 19:00:19.371909 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:19.371885 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw"] Sep 08 19:00:19.377971 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:19.377944 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3adca991-29814900-ll9vw"] Sep 08 19:00:20.025831 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:00:20.025801 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb4b788-e71b-44d5-8f53-8c72d109ebc6" path="/var/lib/kubelet/pods/fbb4b788-e71b-44d5-8f53-8c72d109ebc6/volumes" Sep 08 19:01:05.960834 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.960797 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-e8191e"] Sep 08 19:01:05.961243 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.961204 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1796e83f-d71c-47c2-acf4-bb22e76379e6" containerName="maas-api" Sep 08 19:01:05.961243 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.961219 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1796e83f-d71c-47c2-acf4-bb22e76379e6" containerName="maas-api" Sep 08 19:01:05.961332 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.961248 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb4b788-e71b-44d5-8f53-8c72d109ebc6" containerName="cleanup" Sep 08 19:01:05.961332 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.961255 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb4b788-e71b-44d5-8f53-8c72d109ebc6" containerName="cleanup" Sep 08 19:01:05.961332 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.961315 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb4b788-e71b-44d5-8f53-8c72d109ebc6" containerName="cleanup" Sep 08 19:01:05.961332 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.961324 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1796e83f-d71c-47c2-acf4-bb22e76379e6" containerName="maas-api" Sep 08 19:01:05.964352 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.964334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:05.967105 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.967065 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\"" Sep 08 19:01:05.969777 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:05.969749 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-e8191e"] Sep 08 19:01:06.024347 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:06.024317 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79wp\" (UniqueName: \"kubernetes.io/projected/77092c2b-308c-45f2-ae3d-754478a0ba08-kube-api-access-z79wp\") pod \"test-curl-7695-e8191e\" (UID: \"77092c2b-308c-45f2-ae3d-754478a0ba08\") " pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:06.125193 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:06.125167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79wp\" (UniqueName: \"kubernetes.io/projected/77092c2b-308c-45f2-ae3d-754478a0ba08-kube-api-access-z79wp\") pod \"test-curl-7695-e8191e\" (UID: \"77092c2b-308c-45f2-ae3d-754478a0ba08\") " pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:06.133322 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:06.133300 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79wp\" (UniqueName: \"kubernetes.io/projected/77092c2b-308c-45f2-ae3d-754478a0ba08-kube-api-access-z79wp\") pod \"test-curl-7695-e8191e\" (UID: \"77092c2b-308c-45f2-ae3d-754478a0ba08\") " pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:06.275405 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:06.275377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:06.546868 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:06.546845 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-e8191e"] Sep 08 19:01:06.549190 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:01:06.549164 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77092c2b_308c_45f2_ae3d_754478a0ba08.slice/crio-84c7f73b1414aab4778210e4fb66ec175db166ea382d8d454304b2a8891db08c WatchSource:0}: Error finding container 84c7f73b1414aab4778210e4fb66ec175db166ea382d8d454304b2a8891db08c: Status 404 returned error can't find the container with id 84c7f73b1414aab4778210e4fb66ec175db166ea382d8d454304b2a8891db08c Sep 08 19:01:06.612030 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:06.612001 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-e8191e" event={"ID":"77092c2b-308c-45f2-ae3d-754478a0ba08","Type":"ContainerStarted","Data":"84c7f73b1414aab4778210e4fb66ec175db166ea382d8d454304b2a8891db08c"} Sep 08 19:01:07.618991 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:07.618953 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-e8191e" event={"ID":"77092c2b-308c-45f2-ae3d-754478a0ba08","Type":"ContainerStarted","Data":"3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7"} Sep 08 19:01:07.639040 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:07.638992 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-e8191e" podStartSLOduration=2.638977718 podStartE2EDuration="2.638977718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:01:07.637451606 +0000 UTC m=+1626.146604057" watchObservedRunningTime="2026-09-08 19:01:07.638977718 +0000 UTC m=+1626.148130189" Sep 08 19:01:08.052917 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:08.052885 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl"] Sep 08 19:01:08.053160 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:08.053137 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" containerID="cri-o://3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d" gracePeriod=30 Sep 08 19:01:08.329258 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:08.329176 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-e8191e"] Sep 08 19:01:08.335465 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:08.335441 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-e8191e"] Sep 08 19:01:08.588558 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:08.588474 2557 patch_prober.go:28] interesting pod/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 19:01:08.588558 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:08.588527 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 19:01:08.625805 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:01:08.625776 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-e8191e" container="test-curl-7695-e8191e" Sep 08 19:01:09.631447 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:09.631383 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-e8191e" podUID="77092c2b-308c-45f2-ae3d-754478a0ba08" containerName="test-curl-7695-e8191e" containerID="cri-o://3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7" gracePeriod=2 Sep 08 19:01:09.633484 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:01:09.633451 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-e8191e\" is forbidden: User \"system:node:ip-10-0-134-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-251.ec2.internal' and this object" podUID="77092c2b-308c-45f2-ae3d-754478a0ba08" pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:11.828782 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:11.828753 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:11.977891 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:11.977865 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z79wp\" (UniqueName: \"kubernetes.io/projected/77092c2b-308c-45f2-ae3d-754478a0ba08-kube-api-access-z79wp\") pod \"77092c2b-308c-45f2-ae3d-754478a0ba08\" (UID: \"77092c2b-308c-45f2-ae3d-754478a0ba08\") " Sep 08 19:01:11.979926 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:11.979901 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77092c2b-308c-45f2-ae3d-754478a0ba08-kube-api-access-z79wp" (OuterVolumeSpecName: "kube-api-access-z79wp") pod "77092c2b-308c-45f2-ae3d-754478a0ba08" (UID: "77092c2b-308c-45f2-ae3d-754478a0ba08"). InnerVolumeSpecName "kube-api-access-z79wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:01:12.026554 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.026524 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77092c2b-308c-45f2-ae3d-754478a0ba08" path="/var/lib/kubelet/pods/77092c2b-308c-45f2-ae3d-754478a0ba08/volumes" Sep 08 19:01:12.078604 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.078554 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79wp\" (UniqueName: \"kubernetes.io/projected/77092c2b-308c-45f2-ae3d-754478a0ba08-kube-api-access-z79wp\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:12.648274 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.648234 2557 generic.go:356] "Generic (PLEG): container finished" podID="77092c2b-308c-45f2-ae3d-754478a0ba08" containerID="3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7" exitCode=137 Sep 08 19:01:12.648274 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.648274 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e8191e" Sep 08 19:01:12.648509 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.648297 2557 scope.go:117] "RemoveContainer" containerID="3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7" Sep 08 19:01:12.656749 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.656731 2557 scope.go:117] "RemoveContainer" containerID="3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7" Sep 08 19:01:12.656991 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:01:12.656972 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7\": container with ID starting with 3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7 not found: ID does not exist" containerID="3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7" Sep 08 19:01:12.657042 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:12.656999 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7"} err="failed to get container status \"3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7\": rpc error: code = NotFound desc = could not find container \"3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7\": container with ID starting with 3485283f5a82e2a66ce8a228b3365edbb133d627aa1ea004c5f3038abde42ac7 not found: ID does not exist" Sep 08 19:01:13.309621 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.309594 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 19:01:13.388596 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388528 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-certs\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.388596 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388578 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bce6761-613e-49c9-9f87-d989d199c653-istio-podinfo\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.388827 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388604 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-data\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.388827 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388625 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-istio-token\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.388827 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388688 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-envoy\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.388827 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388719 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-credential-socket\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.388827 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388810 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bce6761-613e-49c9-9f87-d989d199c653-istiod-ca-cert\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.389045 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388866 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-socket\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.389045 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388866 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:01:13.389045 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388908 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgqfw\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-kube-api-access-wgqfw\") pod \"3bce6761-613e-49c9-9f87-d989d199c653\" (UID: \"3bce6761-613e-49c9-9f87-d989d199c653\") " Sep 08 19:01:13.389045 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.388912 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:01:13.389856 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.389657 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bce6761-613e-49c9-9f87-d989d199c653-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:01:13.389856 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.389804 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:01:13.390108 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.390081 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:01:13.390489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.390464 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.390584 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.390494 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.390584 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.390508 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.390584 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.390529 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bce6761-613e-49c9-9f87-d989d199c653-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.390584 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.390542 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.391439 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.391406 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:01:13.392137 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.392101 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bce6761-613e-49c9-9f87-d989d199c653-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:01:13.399777 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.399259 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:01:13.400727 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.400705 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-kube-api-access-wgqfw" (OuterVolumeSpecName: "kube-api-access-wgqfw") pod "3bce6761-613e-49c9-9f87-d989d199c653" (UID: "3bce6761-613e-49c9-9f87-d989d199c653"). InnerVolumeSpecName "kube-api-access-wgqfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:01:13.491184 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.491160 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqfw\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-kube-api-access-wgqfw\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.491184 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.491182 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bce6761-613e-49c9-9f87-d989d199c653-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.491310 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.491191 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bce6761-613e-49c9-9f87-d989d199c653-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.491310 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.491199 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bce6761-613e-49c9-9f87-d989d199c653-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:01:13.654960 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.654875 2557 generic.go:356] "Generic (PLEG): container finished" podID="3bce6761-613e-49c9-9f87-d989d199c653" containerID="3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d" exitCode=0 Sep 08 19:01:13.654960 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.654945 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" Sep 08 19:01:13.654960 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.654953 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" event={"ID":"3bce6761-613e-49c9-9f87-d989d199c653","Type":"ContainerDied","Data":"3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d"} Sep 08 19:01:13.655227 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.655038 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl" event={"ID":"3bce6761-613e-49c9-9f87-d989d199c653","Type":"ContainerDied","Data":"09ac6c906f04a3db6a9aef30935204e939b735e89c537db3c51c8d23746cbd9c"} Sep 08 19:01:13.655227 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.655069 2557 scope.go:117] "RemoveContainer" containerID="3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d" Sep 08 19:01:13.663776 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.663759 2557 scope.go:117] "RemoveContainer" containerID="621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4" Sep 08 19:01:13.671795 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.671772 2557 scope.go:117] "RemoveContainer" containerID="3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d" Sep 08 19:01:13.672043 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:01:13.672024 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d\": container with ID starting with 3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d not found: ID does not exist" containerID="3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d" Sep 08 19:01:13.672134 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.672055 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d"} err="failed to get container status \"3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d\": rpc error: code = NotFound desc = could not find container \"3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d\": container with ID starting with 3ab0ec0fa4219df73a2ca7f064a47d831347f91772743a66f50cef3aee38026d not found: ID does not exist" Sep 08 19:01:13.672134 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.672076 2557 scope.go:117] "RemoveContainer" containerID="621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4" Sep 08 19:01:13.672325 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:01:13.672305 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4\": container with ID starting with 621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4 not found: ID does not exist" containerID="621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4" Sep 08 19:01:13.672373 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.672331 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4"} err="failed to get container status \"621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4\": rpc error: code = NotFound desc = could not find container \"621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4\": container with ID starting with 621f75f1a2637e7b03d144285ea830da5a00a27a09e93bbdaa7e6b70aa4734a4 not found: ID does not exist" Sep 08 19:01:13.676102 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.676081 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl"] Sep 08 19:01:13.682147 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:13.682126 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3adca991-openshift-default-d6c4bf448-h2ttl"] Sep 08 19:01:14.027687 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:01:14.027637 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bce6761-613e-49c9-9f87-d989d199c653" path="/var/lib/kubelet/pods/3bce6761-613e-49c9-9f87-d989d199c653/volumes" Sep 08 19:02:03.330092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.330058 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc"] Sep 08 19:02:03.331186 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331161 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77092c2b-308c-45f2-ae3d-754478a0ba08" containerName="test-curl-7695-e8191e" Sep 08 19:02:03.331186 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331186 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="77092c2b-308c-45f2-ae3d-754478a0ba08" containerName="test-curl-7695-e8191e" Sep 08 19:02:03.331376 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331235 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" Sep 08 19:02:03.331376 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331244 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" Sep 08 19:02:03.331376 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331257 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" Sep 08 19:02:03.331376 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331265 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" Sep 08 19:02:03.331376 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331371 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="77092c2b-308c-45f2-ae3d-754478a0ba08" containerName="test-curl-7695-e8191e" Sep 08 19:02:03.331689 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.331386 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" Sep 08 19:02:03.336367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.336338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.338628 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.338608 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-965746-openshift-default-dockercfg-r4p8f\"" Sep 08 19:02:03.343170 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.343146 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc"] Sep 08 19:02:03.404283 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404255 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-certs\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404283 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d333102-3ece-436e-b3db-69167bfe443f-istio-podinfo\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404302 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-data\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404370 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-credential-socket\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404518 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksrp\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-kube-api-access-2ksrp\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404518 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404419 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-socket\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404608 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404515 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d333102-3ece-436e-b3db-69167bfe443f-istiod-ca-cert\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404608 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404566 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-envoy\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.404608 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.404591 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-istio-token\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.505974 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.505934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-envoy\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506150 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.505984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-istio-token\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506150 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506040 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-certs\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506150 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d333102-3ece-436e-b3db-69167bfe443f-istio-podinfo\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506150 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-data\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506150 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506122 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-credential-socket\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506435 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506247 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksrp\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-kube-api-access-2ksrp\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506435 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506285 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-socket\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506435 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d333102-3ece-436e-b3db-69167bfe443f-istiod-ca-cert\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506597 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-certs\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.506908 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506881 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-data\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.507000 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506903 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-credential-socket\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.507000 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.506935 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-socket\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.507077 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.507012 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d333102-3ece-436e-b3db-69167bfe443f-istiod-ca-cert\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.508801 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.508777 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-envoy\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.509071 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.509050 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d333102-3ece-436e-b3db-69167bfe443f-istio-podinfo\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.514033 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.514007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-istio-token\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.514276 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.514257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksrp\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-kube-api-access-2ksrp\") pod \"e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.649362 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.649280 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:03.946853 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.946718 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc"] Sep 08 19:02:03.949147 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:02:03.949117 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d333102_3ece_436e_b3db_69167bfe443f.slice/crio-5622151b3fff1ff6e2a83c9cf35bd2d87497496eb5ab75232ed81409f660fd04 WatchSource:0}: Error finding container 5622151b3fff1ff6e2a83c9cf35bd2d87497496eb5ab75232ed81409f660fd04: Status 404 returned error can't find the container with id 5622151b3fff1ff6e2a83c9cf35bd2d87497496eb5ab75232ed81409f660fd04 Sep 08 19:02:03.951239 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.951208 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:03.951310 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.951274 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:03.951310 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:03.951302 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:04.930803 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:04.930764 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" event={"ID":"2d333102-3ece-436e-b3db-69167bfe443f","Type":"ContainerStarted","Data":"606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a"} Sep 08 19:02:04.931219 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:04.930822 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" event={"ID":"2d333102-3ece-436e-b3db-69167bfe443f","Type":"ContainerStarted","Data":"5622151b3fff1ff6e2a83c9cf35bd2d87497496eb5ab75232ed81409f660fd04"} Sep 08 19:02:04.953313 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:04.953259 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podStartSLOduration=1.953244337 podStartE2EDuration="1.953244337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:04.950784104 +0000 UTC m=+1683.459936558" watchObservedRunningTime="2026-09-08 19:02:04.953244337 +0000 UTC m=+1683.462396778" Sep 08 19:02:05.650400 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:05.650368 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:05.655110 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:05.655085 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:05.937589 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:05.937517 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:05.938345 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:05.938328 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:08.213781 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.213746 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q"] Sep 08 19:02:08.214459 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.214440 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bce6761-613e-49c9-9f87-d989d199c653" containerName="istio-proxy" Sep 08 19:02:08.217974 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.217948 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.220223 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.220200 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-965746-serving-cert\"" Sep 08 19:02:08.227386 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.227072 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q"] Sep 08 19:02:08.352224 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.352186 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-metrics-certs\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.352502 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.352271 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-maas-api-tls\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.352502 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.352297 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d95ab58-fdf5-4c51-a930-567f05eb9f55-openshift-service-ca\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.352502 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.352326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgd22\" (UniqueName: \"kubernetes.io/projected/2d95ab58-fdf5-4c51-a930-567f05eb9f55-kube-api-access-fgd22\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.453137 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.453104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-maas-api-tls\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.453137 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.453138 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d95ab58-fdf5-4c51-a930-567f05eb9f55-openshift-service-ca\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.453364 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.453161 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgd22\" (UniqueName: \"kubernetes.io/projected/2d95ab58-fdf5-4c51-a930-567f05eb9f55-kube-api-access-fgd22\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.453364 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.453230 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-metrics-certs\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.453825 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.453798 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d95ab58-fdf5-4c51-a930-567f05eb9f55-openshift-service-ca\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.455553 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.455528 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-metrics-certs\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.455662 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.455633 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-maas-api-tls\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.460904 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.460879 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgd22\" (UniqueName: \"kubernetes.io/projected/2d95ab58-fdf5-4c51-a930-567f05eb9f55-kube-api-access-fgd22\") pod \"maas-api-e2e-api-a-965746-576f469d5c-d9f5q\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.531620 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.531595 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:08.804955 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.804931 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q"] Sep 08 19:02:08.806379 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:02:08.806353 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d95ab58_fdf5_4c51_a930_567f05eb9f55.slice/crio-90ec475c9c2b5117a64738b8e0065d6bc629d79627673c77214860fb639f5ee2 WatchSource:0}: Error finding container 90ec475c9c2b5117a64738b8e0065d6bc629d79627673c77214860fb639f5ee2: Status 404 returned error can't find the container with id 90ec475c9c2b5117a64738b8e0065d6bc629d79627673c77214860fb639f5ee2 Sep 08 19:02:08.955496 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:08.955401 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" event={"ID":"2d95ab58-fdf5-4c51-a930-567f05eb9f55","Type":"ContainerStarted","Data":"90ec475c9c2b5117a64738b8e0065d6bc629d79627673c77214860fb639f5ee2"} Sep 08 19:02:09.963327 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:09.963297 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" event={"ID":"2d95ab58-fdf5-4c51-a930-567f05eb9f55","Type":"ContainerStarted","Data":"fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53"} Sep 08 19:02:09.963735 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:09.963359 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:09.979075 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:09.978992 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" podStartSLOduration=1.507641601 podStartE2EDuration="1.978975115s" totalImagesPullingTime="471.333514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:09.978846695 +0000 UTC m=+1688.487999136" watchObservedRunningTime="2026-09-08 19:02:09.978975115 +0000 UTC m=+1688.488127555" Sep 08 19:02:15.974148 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:15.974112 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:02:25.427489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.427393 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-8a443a"] Sep 08 19:02:25.436418 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.435796 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:25.439428 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.439404 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-8a443a"] Sep 08 19:02:25.441725 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.441700 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\"" Sep 08 19:02:25.506024 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.506001 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77v75\" (UniqueName: \"kubernetes.io/projected/a373013b-badb-4643-83e2-c15e841805d3-kube-api-access-77v75\") pod \"test-curl-7695-8a443a\" (UID: \"a373013b-badb-4643-83e2-c15e841805d3\") " pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:25.607096 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.607063 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77v75\" (UniqueName: \"kubernetes.io/projected/a373013b-badb-4643-83e2-c15e841805d3-kube-api-access-77v75\") pod \"test-curl-7695-8a443a\" (UID: \"a373013b-badb-4643-83e2-c15e841805d3\") " pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:25.625338 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.625315 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77v75\" (UniqueName: \"kubernetes.io/projected/a373013b-badb-4643-83e2-c15e841805d3-kube-api-access-77v75\") pod \"test-curl-7695-8a443a\" (UID: \"a373013b-badb-4643-83e2-c15e841805d3\") " pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:25.751342 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:25.751315 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:26.031072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:26.029602 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-8a443a"] Sep 08 19:02:26.057229 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:26.057201 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-8a443a" event={"ID":"a373013b-badb-4643-83e2-c15e841805d3","Type":"ContainerStarted","Data":"2524fac94edcb621a7e78ac7222ebcf208a7b127d8b8c305b05b226cb53a4881"} Sep 08 19:02:27.064434 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:27.064399 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-8a443a" event={"ID":"a373013b-badb-4643-83e2-c15e841805d3","Type":"ContainerStarted","Data":"5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1"} Sep 08 19:02:27.086135 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:27.086092 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-8a443a" podStartSLOduration=2.08605861 podStartE2EDuration="2.08605861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:27.082070963 +0000 UTC m=+1705.591223404" watchObservedRunningTime="2026-09-08 19:02:27.08605861 +0000 UTC m=+1705.595211051" Sep 08 19:02:27.697043 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:27.697009 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-8a443a"] Sep 08 19:02:27.702672 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:27.702363 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-8a443a"] Sep 08 19:02:28.069600 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:02:28.069571 2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-8a443a" container="test-curl-7695-8a443a" Sep 08 19:02:29.075148 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:29.075083 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-8a443a" podUID="a373013b-badb-4643-83e2-c15e841805d3" containerName="test-curl-7695-8a443a" containerID="cri-o://5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1" gracePeriod=2 Sep 08 19:02:31.086257 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.086231 2557 generic.go:356] "Generic (PLEG): container finished" podID="2d333102-3ece-436e-b3db-69167bfe443f" containerID="606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a" exitCode=0 Sep 08 19:02:31.086640 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.086299 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" event={"ID":"2d333102-3ece-436e-b3db-69167bfe443f","Type":"ContainerDied","Data":"606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a"} Sep 08 19:02:31.086795 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.086778 2557 scope.go:117] "RemoveContainer" containerID="606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a" Sep 08 19:02:31.087862 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.087838 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:31.087939 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.087900 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:31.087939 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.087932 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:31.233338 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.233315 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:31.277199 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.277175 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77v75\" (UniqueName: \"kubernetes.io/projected/a373013b-badb-4643-83e2-c15e841805d3-kube-api-access-77v75\") pod \"a373013b-badb-4643-83e2-c15e841805d3\" (UID: \"a373013b-badb-4643-83e2-c15e841805d3\") " Sep 08 19:02:31.279342 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.279313 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a373013b-badb-4643-83e2-c15e841805d3-kube-api-access-77v75" (OuterVolumeSpecName: "kube-api-access-77v75") pod "a373013b-badb-4643-83e2-c15e841805d3" (UID: "a373013b-badb-4643-83e2-c15e841805d3"). InnerVolumeSpecName "kube-api-access-77v75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:02:31.377839 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:31.377805 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77v75\" (UniqueName: \"kubernetes.io/projected/a373013b-badb-4643-83e2-c15e841805d3-kube-api-access-77v75\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:02:32.030504 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.030458 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a373013b-badb-4643-83e2-c15e841805d3" path="/var/lib/kubelet/pods/a373013b-badb-4643-83e2-c15e841805d3/volumes" Sep 08 19:02:32.092499 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.092470 2557 generic.go:356] "Generic (PLEG): container finished" podID="a373013b-badb-4643-83e2-c15e841805d3" containerID="5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1" exitCode=137 Sep 08 19:02:32.092956 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.092512 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-8a443a" Sep 08 19:02:32.092956 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.092554 2557 scope.go:117] "RemoveContainer" containerID="5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1" Sep 08 19:02:32.094965 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.094943 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" event={"ID":"2d333102-3ece-436e-b3db-69167bfe443f","Type":"ContainerStarted","Data":"6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4"} Sep 08 19:02:32.101336 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.101321 2557 scope.go:117] "RemoveContainer" containerID="5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1" Sep 08 19:02:32.101604 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:02:32.101589 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1\": container with ID starting with 5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1 not found: ID does not exist" containerID="5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1" Sep 08 19:02:32.101669 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.101611 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1"} err="failed to get container status \"5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1\": rpc error: code = NotFound desc = could not find container \"5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1\": container with ID starting with 5068bf6def290d5f0625c60381b6623c2108ec472e8d301db527d7d830fd7ad1 not found: ID does not exist" Sep 08 19:02:32.651669 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.651142 2557 patch_prober.go:28] interesting pod/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 08 19:02:32.651669 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.651208 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:32.651669 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.651485 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 08 19:02:32.654689 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.652301 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt"] Sep 08 19:02:32.654689 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.653112 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a373013b-badb-4643-83e2-c15e841805d3" containerName="test-curl-7695-8a443a" Sep 08 19:02:32.654689 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.653134 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a373013b-badb-4643-83e2-c15e841805d3" containerName="test-curl-7695-8a443a" Sep 08 19:02:32.654689 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.653265 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a373013b-badb-4643-83e2-c15e841805d3" containerName="test-curl-7695-8a443a" Sep 08 19:02:32.671087 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.671057 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt"] Sep 08 19:02:32.671279 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.671184 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.674291 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.674268 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ee8372-openshift-default-dockercfg-k7wvk\"" Sep 08 19:02:32.689978 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.689946 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715fa630-9961-4811-807a-506d00182b75-istiod-ca-cert\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690098 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690035 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715fa630-9961-4811-807a-506d00182b75-istio-podinfo\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690098 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690090 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-istio-token\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690233 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690152 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-certs\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690233 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690177 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-data\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690233 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690193 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-socket\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690397 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690320 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-credential-socket\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690397 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-envoy\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.690397 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.690373 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbl4\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-kube-api-access-lzbl4\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.791826 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.791796 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-certs\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.791996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.791832 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-data\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.791996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.791851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-socket\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.791996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.791918 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-credential-socket\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.791996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.791934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-envoy\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.791996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.791952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbl4\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-kube-api-access-lzbl4\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792272 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-certs\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792317 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792280 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-socket\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792317 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715fa630-9961-4811-807a-506d00182b75-istiod-ca-cert\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792393 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715fa630-9961-4811-807a-506d00182b75-istio-podinfo\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792393 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792309 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-data\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792393 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792375 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-credential-socket\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792535 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-istio-token\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.792933 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.792904 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715fa630-9961-4811-807a-506d00182b75-istiod-ca-cert\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.794346 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.794326 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-envoy\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.794760 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.794739 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715fa630-9961-4811-807a-506d00182b75-istio-podinfo\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.805708 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.805682 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbl4\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-kube-api-access-lzbl4\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.826671 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.826059 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-istio-token\") pod \"e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:32.982635 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:32.982603 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:33.295604 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:33.295577 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt"] Sep 08 19:02:33.298019 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:02:33.297995 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715fa630_9961_4811_807a_506d00182b75.slice/crio-ceb837f667d5980f3d923cb60635bd160dcc85af882bd6e2cb472f235113e77a WatchSource:0}: Error finding container ceb837f667d5980f3d923cb60635bd160dcc85af882bd6e2cb472f235113e77a: Status 404 returned error can't find the container with id ceb837f667d5980f3d923cb60635bd160dcc85af882bd6e2cb472f235113e77a Sep 08 19:02:33.300502 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:33.300469 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:33.300600 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:33.300547 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:33.300600 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:33.300590 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:33.649950 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:33.649873 2557 patch_prober.go:28] interesting pod/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 08 19:02:33.650433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:33.650142 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 08 19:02:34.109713 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:34.109671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" event={"ID":"715fa630-9961-4811-807a-506d00182b75","Type":"ContainerStarted","Data":"ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787"} Sep 08 19:02:34.109902 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:34.109743 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" event={"ID":"715fa630-9961-4811-807a-506d00182b75","Type":"ContainerStarted","Data":"ceb837f667d5980f3d923cb60635bd160dcc85af882bd6e2cb472f235113e77a"} Sep 08 19:02:34.650639 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:34.650605 2557 patch_prober.go:28] interesting pod/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 08 19:02:34.651043 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:34.650678 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 08 19:02:34.983518 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:34.983492 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:34.988763 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:34.988742 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:35.014469 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:35.014415 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podStartSLOduration=3.014396154 podStartE2EDuration="3.014396154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:34.132843137 +0000 UTC m=+1712.641995578" watchObservedRunningTime="2026-09-08 19:02:35.014396154 +0000 UTC m=+1713.523548598" Sep 08 19:02:35.115909 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:35.115883 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:35.117158 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:35.116939 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:35.650004 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:35.649961 2557 patch_prober.go:28] interesting pod/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 08 19:02:35.650178 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:35.650024 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 08 19:02:35.939066 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:35.938170 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:36.650505 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:36.650470 2557 patch_prober.go:28] interesting pod/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 08 19:02:36.650697 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:36.650528 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 08 19:02:37.653912 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:37.653869 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:37.655152 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:37.655130 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:02:56.241258 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:56.241229 2557 generic.go:356] "Generic (PLEG): container finished" podID="715fa630-9961-4811-807a-506d00182b75" containerID="ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787" exitCode=0 Sep 08 19:02:56.241700 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:56.241300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" event={"ID":"715fa630-9961-4811-807a-506d00182b75","Type":"ContainerDied","Data":"ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787"} Sep 08 19:02:56.241700 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:56.241634 2557 scope.go:117] "RemoveContainer" containerID="ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787" Sep 08 19:02:56.242743 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:56.242716 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:56.242795 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:56.242783 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:56.242838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:56.242832 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:02:57.250335 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:57.250298 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" event={"ID":"715fa630-9961-4811-807a-506d00182b75","Type":"ContainerStarted","Data":"d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078"} Sep 08 19:02:57.982866 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:57.982835 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:02:57.983038 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:57.982997 2557 patch_prober.go:28] interesting pod/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 08 19:02:57.983106 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:57.983073 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 08 19:02:58.983728 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:58.983689 2557 patch_prober.go:28] interesting pod/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 08 19:02:58.984107 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:58.983753 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 08 19:02:59.983502 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:59.983465 2557 patch_prober.go:28] interesting pod/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 08 19:02:59.983700 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:02:59.983518 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 08 19:03:00.983437 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:00.983400 2557 patch_prober.go:28] interesting pod/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 08 19:03:00.983841 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:00.983478 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 08 19:03:01.983824 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:01.983784 2557 patch_prober.go:28] interesting pod/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 08 19:03:01.984251 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:01.984096 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 08 19:03:02.986892 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:02.986862 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:03:02.987325 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:02.987150 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:03:02.987918 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:02.987900 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:03:19.477259 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.477224 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q"] Sep 08 19:03:19.477735 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.477491 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" podUID="2d95ab58-fdf5-4c51-a930-567f05eb9f55" containerName="maas-api" containerID="cri-o://fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53" gracePeriod=30 Sep 08 19:03:19.735744 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.735689 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:03:19.838735 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.838695 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-maas-api-tls\") pod \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " Sep 08 19:03:19.838917 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.838777 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d95ab58-fdf5-4c51-a930-567f05eb9f55-openshift-service-ca\") pod \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " Sep 08 19:03:19.838917 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.838805 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgd22\" (UniqueName: \"kubernetes.io/projected/2d95ab58-fdf5-4c51-a930-567f05eb9f55-kube-api-access-fgd22\") pod \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " Sep 08 19:03:19.838917 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.838828 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-metrics-certs\") pod \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\" (UID: \"2d95ab58-fdf5-4c51-a930-567f05eb9f55\") " Sep 08 19:03:19.839191 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.839163 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d95ab58-fdf5-4c51-a930-567f05eb9f55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d95ab58-fdf5-4c51-a930-567f05eb9f55" (UID: "2d95ab58-fdf5-4c51-a930-567f05eb9f55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:19.840996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.840963 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d95ab58-fdf5-4c51-a930-567f05eb9f55-kube-api-access-fgd22" (OuterVolumeSpecName: "kube-api-access-fgd22") pod "2d95ab58-fdf5-4c51-a930-567f05eb9f55" (UID: "2d95ab58-fdf5-4c51-a930-567f05eb9f55"). InnerVolumeSpecName "kube-api-access-fgd22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:19.841133 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.840990 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2d95ab58-fdf5-4c51-a930-567f05eb9f55" (UID: "2d95ab58-fdf5-4c51-a930-567f05eb9f55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:03:19.841133 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.841002 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d95ab58-fdf5-4c51-a930-567f05eb9f55" (UID: "2d95ab58-fdf5-4c51-a930-567f05eb9f55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:03:19.940152 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.940120 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:19.940152 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.940150 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d95ab58-fdf5-4c51-a930-567f05eb9f55-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:19.940367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.940199 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgd22\" (UniqueName: \"kubernetes.io/projected/2d95ab58-fdf5-4c51-a930-567f05eb9f55-kube-api-access-fgd22\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:19.940367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:19.940209 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d95ab58-fdf5-4c51-a930-567f05eb9f55-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:20.380766 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.380689 2557 generic.go:356] "Generic (PLEG): container finished" podID="2d95ab58-fdf5-4c51-a930-567f05eb9f55" containerID="fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53" exitCode=0 Sep 08 19:03:20.380766 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.380730 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" Sep 08 19:03:20.380943 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.380773 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" event={"ID":"2d95ab58-fdf5-4c51-a930-567f05eb9f55","Type":"ContainerDied","Data":"fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53"} Sep 08 19:03:20.380943 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.380822 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q" event={"ID":"2d95ab58-fdf5-4c51-a930-567f05eb9f55","Type":"ContainerDied","Data":"90ec475c9c2b5117a64738b8e0065d6bc629d79627673c77214860fb639f5ee2"} Sep 08 19:03:20.380943 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.380843 2557 scope.go:117] "RemoveContainer" containerID="fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53" Sep 08 19:03:20.389775 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.389755 2557 scope.go:117] "RemoveContainer" containerID="fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53" Sep 08 19:03:20.390046 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:03:20.390026 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53\": container with ID starting with fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53 not found: ID does not exist" containerID="fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53" Sep 08 19:03:20.390093 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.390054 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53"} err="failed to get container status \"fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53\": rpc error: code = NotFound desc = could not find container \"fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53\": container with ID starting with fb4cd11f087f521291ae10704d7ff11711b8ace8e4c99dab21ee7bd50bac8b53 not found: ID does not exist" Sep 08 19:03:20.397174 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.397153 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q"] Sep 08 19:03:20.401230 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:20.401209 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-965746-576f469d5c-d9f5q"] Sep 08 19:03:22.026611 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:22.026577 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d95ab58-fdf5-4c51-a930-567f05eb9f55" path="/var/lib/kubelet/pods/2d95ab58-fdf5-4c51-a930-567f05eb9f55/volumes" Sep 08 19:03:26.177364 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:26.177333 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc"] Sep 08 19:03:26.177832 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:26.177535 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" containerID="cri-o://6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4" gracePeriod=30 Sep 08 19:03:31.436086 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.436059 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:03:31.443322 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.443257 2557 generic.go:356] "Generic (PLEG): container finished" podID="2d333102-3ece-436e-b3db-69167bfe443f" containerID="6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4" exitCode=0 Sep 08 19:03:31.443322 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.443315 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" Sep 08 19:03:31.443469 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.443299 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" event={"ID":"2d333102-3ece-436e-b3db-69167bfe443f","Type":"ContainerDied","Data":"6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4"} Sep 08 19:03:31.443469 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.443436 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc" event={"ID":"2d333102-3ece-436e-b3db-69167bfe443f","Type":"ContainerDied","Data":"5622151b3fff1ff6e2a83c9cf35bd2d87497496eb5ab75232ed81409f660fd04"} Sep 08 19:03:31.443469 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.443457 2557 scope.go:117] "RemoveContainer" containerID="6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4" Sep 08 19:03:31.452221 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.452202 2557 scope.go:117] "RemoveContainer" containerID="606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a" Sep 08 19:03:31.462163 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.462145 2557 scope.go:117] "RemoveContainer" containerID="6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4" Sep 08 19:03:31.462456 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:03:31.462433 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4\": container with ID starting with 6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4 not found: ID does not exist" containerID="6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4" Sep 08 19:03:31.462563 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.462468 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4"} err="failed to get container status \"6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4\": rpc error: code = NotFound desc = could not find container \"6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4\": container with ID starting with 6caefacd74394a28260c9ede2a02b465a2a375901bbbc83edecc9bf7cb558ec4 not found: ID does not exist" Sep 08 19:03:31.462563 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.462492 2557 scope.go:117] "RemoveContainer" containerID="606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a" Sep 08 19:03:31.462845 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:03:31.462822 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a\": container with ID starting with 606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a not found: ID does not exist" containerID="606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a" Sep 08 19:03:31.462920 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.462864 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a"} err="failed to get container status \"606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a\": rpc error: code = NotFound desc = could not find container \"606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a\": container with ID starting with 606eddc7b790784d4fdc185a969a38ca3cc22cfbbf23a52defad9da916f2b23a not found: ID does not exist" Sep 08 19:03:31.539181 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539156 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-certs\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539277 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539198 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ksrp\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-kube-api-access-2ksrp\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539277 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539234 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-socket\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539356 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539283 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-envoy\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539356 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539322 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-credential-socket\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539460 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539360 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d333102-3ece-436e-b3db-69167bfe443f-istio-podinfo\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539460 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539393 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-data\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539460 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539426 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d333102-3ece-436e-b3db-69167bfe443f-istiod-ca-cert\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539619 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539487 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-istio-token\") pod \"2d333102-3ece-436e-b3db-69167bfe443f\" (UID: \"2d333102-3ece-436e-b3db-69167bfe443f\") " Sep 08 19:03:31.539619 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539554 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:31.539759 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539688 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:31.539759 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539710 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:31.539867 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539768 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:31.539931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539872 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.539931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539885 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.539931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539890 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d333102-3ece-436e-b3db-69167bfe443f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:31.539931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539899 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.540079 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.539941 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.541431 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.541408 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:31.541555 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.541540 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:31.541816 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.541796 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-kube-api-access-2ksrp" (OuterVolumeSpecName: "kube-api-access-2ksrp") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "kube-api-access-2ksrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:31.541816 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.541802 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d333102-3ece-436e-b3db-69167bfe443f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d333102-3ece-436e-b3db-69167bfe443f" (UID: "2d333102-3ece-436e-b3db-69167bfe443f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:03:31.640698 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.640675 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d333102-3ece-436e-b3db-69167bfe443f-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.640802 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.640701 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.640802 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.640714 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ksrp\" (UniqueName: \"kubernetes.io/projected/2d333102-3ece-436e-b3db-69167bfe443f-kube-api-access-2ksrp\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.640802 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.640727 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d333102-3ece-436e-b3db-69167bfe443f-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.640802 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.640739 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d333102-3ece-436e-b3db-69167bfe443f-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:31.766704 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.766678 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc"] Sep 08 19:03:31.774270 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:31.774248 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-965746-openshift-default-c64dc8db6-tt7jc"] Sep 08 19:03:32.028889 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:32.028811 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d333102-3ece-436e-b3db-69167bfe443f" path="/var/lib/kubelet/pods/2d333102-3ece-436e-b3db-69167bfe443f/volumes" Sep 08 19:03:48.878599 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:48.878565 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt"] Sep 08 19:03:48.879067 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:48.878874 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" containerID="cri-o://d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078" gracePeriod=30 Sep 08 19:03:54.118796 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.118773 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:03:54.143240 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143162 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbl4\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-kube-api-access-lzbl4\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143240 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143219 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715fa630-9961-4811-807a-506d00182b75-istio-podinfo\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143426 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143282 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-istio-token\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143426 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143319 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-data\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143426 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143370 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715fa630-9961-4811-807a-506d00182b75-istiod-ca-cert\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143426 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143404 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-envoy\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143677 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143430 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-certs\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143677 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143464 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-credential-socket\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.143677 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.143515 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-socket\") pod \"715fa630-9961-4811-807a-506d00182b75\" (UID: \"715fa630-9961-4811-807a-506d00182b75\") " Sep 08 19:03:54.144085 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.144030 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:54.144409 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.144187 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715fa630-9961-4811-807a-506d00182b75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:54.145329 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.145130 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:54.145329 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.145303 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-data" (OuterVolumeSpecName: "istio-data") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:54.145548 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.145521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:54.146091 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.146063 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-kube-api-access-lzbl4" (OuterVolumeSpecName: "kube-api-access-lzbl4") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "kube-api-access-lzbl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:54.146748 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.146721 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/715fa630-9961-4811-807a-506d00182b75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:03:54.147192 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.147158 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:03:54.147584 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.147562 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-istio-token" (OuterVolumeSpecName: "istio-token") pod "715fa630-9961-4811-807a-506d00182b75" (UID: "715fa630-9961-4811-807a-506d00182b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:54.244902 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244872 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.244902 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244895 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.244902 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244904 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715fa630-9961-4811-807a-506d00182b75-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.245092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244912 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.245092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244920 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.245092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244929 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.245092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244936 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715fa630-9961-4811-807a-506d00182b75-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.245092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244945 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbl4\" (UniqueName: \"kubernetes.io/projected/715fa630-9961-4811-807a-506d00182b75-kube-api-access-lzbl4\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.245092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.244953 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715fa630-9961-4811-807a-506d00182b75-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:03:54.567606 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.567575 2557 generic.go:356] "Generic (PLEG): container finished" podID="715fa630-9961-4811-807a-506d00182b75" containerID="d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078" exitCode=0 Sep 08 19:03:54.567786 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.567621 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" Sep 08 19:03:54.567786 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.567679 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" event={"ID":"715fa630-9961-4811-807a-506d00182b75","Type":"ContainerDied","Data":"d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078"} Sep 08 19:03:54.567786 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.567732 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt" event={"ID":"715fa630-9961-4811-807a-506d00182b75","Type":"ContainerDied","Data":"ceb837f667d5980f3d923cb60635bd160dcc85af882bd6e2cb472f235113e77a"} Sep 08 19:03:54.567786 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.567754 2557 scope.go:117] "RemoveContainer" containerID="d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078" Sep 08 19:03:54.577175 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.577159 2557 scope.go:117] "RemoveContainer" containerID="ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787" Sep 08 19:03:54.585111 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.585094 2557 scope.go:117] "RemoveContainer" containerID="d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078" Sep 08 19:03:54.585342 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:03:54.585326 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078\": container with ID starting with d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078 not found: ID does not exist" containerID="d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078" Sep 08 19:03:54.585392 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.585352 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078"} err="failed to get container status \"d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078\": rpc error: code = NotFound desc = could not find container \"d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078\": container with ID starting with d48841acda72c39511f763def704ad48225c5f4dc5e28c81d942ce5035893078 not found: ID does not exist" Sep 08 19:03:54.585392 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.585373 2557 scope.go:117] "RemoveContainer" containerID="ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787" Sep 08 19:03:54.585703 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:03:54.585675 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787\": container with ID starting with ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787 not found: ID does not exist" containerID="ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787" Sep 08 19:03:54.585817 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.585712 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787"} err="failed to get container status \"ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787\": rpc error: code = NotFound desc = could not find container \"ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787\": container with ID starting with ae6bf96db72987007173d39b9725a940db5c9cdd8e65772835579bb4eee6b787 not found: ID does not exist" Sep 08 19:03:54.588235 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.588216 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt"] Sep 08 19:03:54.592048 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:54.592028 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ee8372-openshift-default-755c799b8b-g9bmt"] Sep 08 19:03:56.026659 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:56.026619 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="715fa630-9961-4811-807a-506d00182b75" path="/var/lib/kubelet/pods/715fa630-9961-4811-807a-506d00182b75/volumes" Sep 08 19:03:59.416373 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.416339 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k"] Sep 08 19:03:59.416963 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.416943 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d95ab58-fdf5-4c51-a930-567f05eb9f55" containerName="maas-api" Sep 08 19:03:59.417072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.416965 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d95ab58-fdf5-4c51-a930-567f05eb9f55" containerName="maas-api" Sep 08 19:03:59.417072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.416995 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" Sep 08 19:03:59.417072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417004 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" Sep 08 19:03:59.417072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417034 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" Sep 08 19:03:59.417072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417048 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" Sep 08 19:03:59.417072 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417068 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" Sep 08 19:03:59.417367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417076 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" Sep 08 19:03:59.417367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417188 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d95ab58-fdf5-4c51-a930-567f05eb9f55" containerName="maas-api" Sep 08 19:03:59.417367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417203 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" Sep 08 19:03:59.417367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417215 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" Sep 08 19:03:59.417367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417227 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" Sep 08 19:03:59.417367 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.417237 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d333102-3ece-436e-b3db-69167bfe443f" containerName="istio-proxy" Sep 08 19:03:59.422156 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.422132 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.424843 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.424819 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e77525-openshift-default-dockercfg-zfjnt\"" Sep 08 19:03:59.432922 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.432899 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k"] Sep 08 19:03:59.489195 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489166 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-certs\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489319 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489218 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-credential-socket\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489319 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489248 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-envoy\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489319 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-token\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489350 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxq6j\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-kube-api-access-mxq6j\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489372 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-socket\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2122c3-6e99-4db7-959f-66db7ad111f6-istiod-ca-cert\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489527 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489422 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-data\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.489527 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.489485 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-podinfo\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.590687 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-data\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.590872 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590696 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-podinfo\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.590872 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590721 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-certs\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.590872 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-credential-socket\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.590872 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-envoy\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.590872 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-token\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591146 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590896 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxq6j\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-kube-api-access-mxq6j\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591146 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590924 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-socket\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591146 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.590967 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2122c3-6e99-4db7-959f-66db7ad111f6-istiod-ca-cert\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591738 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.591149 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-certs\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591738 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.591278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-credential-socket\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591738 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.591358 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-data\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591738 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.591521 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-socket\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.591945 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.591835 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2122c3-6e99-4db7-959f-66db7ad111f6-istiod-ca-cert\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.593337 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.593314 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-podinfo\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.593470 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.593339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-envoy\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.598161 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.598134 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-token\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.599004 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.598985 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxq6j\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-kube-api-access-mxq6j\") pod \"e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:03:59.734369 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:03:59.734344 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:00.011045 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.011020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k"] Sep 08 19:04:00.013868 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:04:00.013843 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2122c3_6e99_4db7_959f_66db7ad111f6.slice/crio-143054217f51be9839a26296ac91b96b07c6fe548c6a595423a855f63486b4f6 WatchSource:0}: Error finding container 143054217f51be9839a26296ac91b96b07c6fe548c6a595423a855f63486b4f6: Status 404 returned error can't find the container with id 143054217f51be9839a26296ac91b96b07c6fe548c6a595423a855f63486b4f6 Sep 08 19:04:00.016629 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.016588 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:00.016733 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.016703 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:00.016781 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.016747 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:00.046542 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.046408 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz"] Sep 08 19:04:00.046820 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.046638 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" podUID="1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" containerName="maas-api" containerID="cri-o://5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e" gracePeriod=30 Sep 08 19:04:00.308993 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.308964 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 19:04:00.399518 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.399489 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-openshift-service-ca\") pod \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " Sep 08 19:04:00.399518 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.399522 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5qxs\" (UniqueName: \"kubernetes.io/projected/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-kube-api-access-b5qxs\") pod \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " Sep 08 19:04:00.399794 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.399566 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls\") pod \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " Sep 08 19:04:00.399794 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.399713 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-metrics-certs\") pod \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\" (UID: \"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e\") " Sep 08 19:04:00.400000 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.399962 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" (UID: "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:04:00.400145 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.400108 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:00.402234 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.402207 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" (UID: "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:04:00.402336 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.402214 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-kube-api-access-b5qxs" (OuterVolumeSpecName: "kube-api-access-b5qxs") pod "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" (UID: "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e"). InnerVolumeSpecName "kube-api-access-b5qxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:00.402410 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.402337 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" (UID: "1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:04:00.501046 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.501010 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qxs\" (UniqueName: \"kubernetes.io/projected/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-kube-api-access-b5qxs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:00.501046 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.501040 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:00.501046 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.501049 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:00.610096 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.610060 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" event={"ID":"fc2122c3-6e99-4db7-959f-66db7ad111f6","Type":"ContainerStarted","Data":"30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1"} Sep 08 19:04:00.610233 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.610153 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" event={"ID":"fc2122c3-6e99-4db7-959f-66db7ad111f6","Type":"ContainerStarted","Data":"143054217f51be9839a26296ac91b96b07c6fe548c6a595423a855f63486b4f6"} Sep 08 19:04:00.611511 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.611483 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" containerID="5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e" exitCode=0 Sep 08 19:04:00.611613 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.611520 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" Sep 08 19:04:00.611613 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.611561 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" event={"ID":"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e","Type":"ContainerDied","Data":"5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e"} Sep 08 19:04:00.611613 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.611610 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz" event={"ID":"1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e","Type":"ContainerDied","Data":"2c241bd5ea7afdf6934125335c2efbee94013941d323af97f1ccc1b93b474891"} Sep 08 19:04:00.611752 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.611638 2557 scope.go:117] "RemoveContainer" containerID="5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e" Sep 08 19:04:00.622935 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.622807 2557 scope.go:117] "RemoveContainer" containerID="5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e" Sep 08 19:04:00.623113 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:04:00.623087 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e\": container with ID starting with 5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e not found: ID does not exist" containerID="5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e" Sep 08 19:04:00.623185 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.623124 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e"} err="failed to get container status \"5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e\": rpc error: code = NotFound desc = could not find container \"5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e\": container with ID starting with 5236f51f783a5edaad541e4d079daafaf9188398718527d1e19686771a69e01e not found: ID does not exist" Sep 08 19:04:00.630581 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.630541 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podStartSLOduration=1.630530434 podStartE2EDuration="1.630530434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:00.627075337 +0000 UTC m=+1799.136227780" watchObservedRunningTime="2026-09-08 19:04:00.630530434 +0000 UTC m=+1799.139682875" Sep 08 19:04:00.641715 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.641680 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz"] Sep 08 19:04:00.644867 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.644841 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-093570-cbddd5cc9-kfgzz"] Sep 08 19:04:00.734952 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:00.734923 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:01.739757 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:01.739731 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:02.029290 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:02.029209 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" path="/var/lib/kubelet/pods/1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e/volumes" Sep 08 19:04:02.626517 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:02.626484 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:02.627455 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:02.627435 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:03.772834 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.772806 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj"] Sep 08 19:04:03.773363 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.773344 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" containerName="maas-api" Sep 08 19:04:03.773363 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.773365 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" containerName="maas-api" Sep 08 19:04:03.773500 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.773379 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" Sep 08 19:04:03.773500 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.773384 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="715fa630-9961-4811-807a-506d00182b75" containerName="istio-proxy" Sep 08 19:04:03.773500 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.773470 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0fe8ea-54a7-4ab5-af9a-3f20a70e875e" containerName="maas-api" Sep 08 19:04:03.778152 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.778130 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.780228 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.780205 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e77525-serving-cert\"" Sep 08 19:04:03.787822 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.787799 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj"] Sep 08 19:04:03.834932 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.834901 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0607c7ea-8c0e-4d77-9979-57e8e5068937-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.835044 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.834947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22z5b\" (UniqueName: \"kubernetes.io/projected/0607c7ea-8c0e-4d77-9979-57e8e5068937-kube-api-access-22z5b\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.835044 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.834976 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-metrics-certs\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.835123 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.835106 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.935822 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.935789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-metrics-certs\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.936014 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.935872 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.936014 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.935938 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0607c7ea-8c0e-4d77-9979-57e8e5068937-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.936014 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.935979 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22z5b\" (UniqueName: \"kubernetes.io/projected/0607c7ea-8c0e-4d77-9979-57e8e5068937-kube-api-access-22z5b\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.936192 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:04:03.936016 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-serving-cert: secret "maas-api-e2e-ipp-a-e77525-serving-cert" not found Sep 08 19:04:03.936192 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:04:03.936098 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls podName:0607c7ea-8c0e-4d77-9979-57e8e5068937 nodeName:}" failed. No retries permitted until 2026-09-08 19:04:04.43607542 +0000 UTC m=+1802.945227852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls") pod "maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" (UID: "0607c7ea-8c0e-4d77-9979-57e8e5068937") : secret "maas-api-e2e-ipp-a-e77525-serving-cert" not found Sep 08 19:04:03.936592 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.936570 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0607c7ea-8c0e-4d77-9979-57e8e5068937-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.938258 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.938235 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-metrics-certs\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:03.946700 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:03.946680 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22z5b\" (UniqueName: \"kubernetes.io/projected/0607c7ea-8c0e-4d77-9979-57e8e5068937-kube-api-access-22z5b\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:04.441066 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:04.441020 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:04.443468 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:04.443435 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:04.692559 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:04.692474 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:04.966050 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:04.966025 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj"] Sep 08 19:04:04.968012 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:04:04.967983 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0607c7ea_8c0e_4d77_9979_57e8e5068937.slice/crio-fab4b6d86c72e631b3c586e5647c91dbeceefb09a39d54e4b47288d8015309eb WatchSource:0}: Error finding container fab4b6d86c72e631b3c586e5647c91dbeceefb09a39d54e4b47288d8015309eb: Status 404 returned error can't find the container with id fab4b6d86c72e631b3c586e5647c91dbeceefb09a39d54e4b47288d8015309eb Sep 08 19:04:04.969436 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:04.969420 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 19:04:05.644047 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:05.644012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" event={"ID":"0607c7ea-8c0e-4d77-9979-57e8e5068937","Type":"ContainerStarted","Data":"9b4d042f4872e93065836717765852c8919e43f4c22b53c5e75cb275e6b8935b"} Sep 08 19:04:05.644211 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:05.644063 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" event={"ID":"0607c7ea-8c0e-4d77-9979-57e8e5068937","Type":"ContainerStarted","Data":"fab4b6d86c72e631b3c586e5647c91dbeceefb09a39d54e4b47288d8015309eb"} Sep 08 19:04:05.644211 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:05.644094 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:05.660235 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:05.660165 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" podStartSLOduration=2.128099272 podStartE2EDuration="2.660150176s" totalImagesPullingTime="532.050904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:05.659451053 +0000 UTC m=+1804.168603497" watchObservedRunningTime="2026-09-08 19:04:05.660150176 +0000 UTC m=+1804.169302616" Sep 08 19:04:06.752743 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:06.752709 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd"] Sep 08 19:04:06.753220 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:06.752938 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" containerID="cri-o://856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948" gracePeriod=30 Sep 08 19:04:11.654959 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:11.654924 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:04:12.051441 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.051415 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 19:04:12.217575 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217488 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-certs\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217575 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217539 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-socket\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217575 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217572 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-token\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217601 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf65g\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-kube-api-access-nf65g\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217641 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-data\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217717 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13616b0-a2bc-4578-824b-65a4114a96c7-istiod-ca-cert\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217739 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-envoy\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217791 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-podinfo\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.217886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.217843 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-credential-socket\") pod \"c13616b0-a2bc-4578-824b-65a4114a96c7\" (UID: \"c13616b0-a2bc-4578-824b-65a4114a96c7\") " Sep 08 19:04:12.218384 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.218353 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:12.218573 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.218549 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:12.218735 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.218712 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:12.219250 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.219218 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:12.219347 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.219248 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13616b0-a2bc-4578-824b-65a4114a96c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:04:12.231667 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.228749 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:04:12.231667 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.228776 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:12.231667 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.228861 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-kube-api-access-nf65g" (OuterVolumeSpecName: "kube-api-access-nf65g") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "kube-api-access-nf65g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:12.231667 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.230037 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c13616b0-a2bc-4578-824b-65a4114a96c7" (UID: "c13616b0-a2bc-4578-824b-65a4114a96c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:12.319392 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319357 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319392 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319392 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319407 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319426 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf65g\" (UniqueName: \"kubernetes.io/projected/c13616b0-a2bc-4578-824b-65a4114a96c7-kube-api-access-nf65g\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319439 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319452 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13616b0-a2bc-4578-824b-65a4114a96c7-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319465 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319477 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13616b0-a2bc-4578-824b-65a4114a96c7-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.319583 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.319491 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13616b0-a2bc-4578-824b-65a4114a96c7-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:04:12.682772 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.682741 2557 generic.go:356] "Generic (PLEG): container finished" podID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerID="856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948" exitCode=0 Sep 08 19:04:12.683151 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.682821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" event={"ID":"c13616b0-a2bc-4578-824b-65a4114a96c7","Type":"ContainerDied","Data":"856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948"} Sep 08 19:04:12.683151 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.682875 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" event={"ID":"c13616b0-a2bc-4578-824b-65a4114a96c7","Type":"ContainerDied","Data":"12dcff5dc255319453b0b4c7011b82bac49836b4d755d7e33782eb8059c5548f"} Sep 08 19:04:12.683151 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.682897 2557 scope.go:117] "RemoveContainer" containerID="856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948" Sep 08 19:04:12.683318 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.683195 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd" Sep 08 19:04:12.693341 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.693323 2557 scope.go:117] "RemoveContainer" containerID="8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5" Sep 08 19:04:12.702387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.702350 2557 scope.go:117] "RemoveContainer" containerID="856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948" Sep 08 19:04:12.702769 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:04:12.702750 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948\": container with ID starting with 856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948 not found: ID does not exist" containerID="856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948" Sep 08 19:04:12.702853 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.702777 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948"} err="failed to get container status \"856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948\": rpc error: code = NotFound desc = could not find container \"856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948\": container with ID starting with 856d89e69be2b1027e12a11d74de54167f9240b09773da38481809c3fdd4b948 not found: ID does not exist" Sep 08 19:04:12.702853 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.702792 2557 scope.go:117] "RemoveContainer" containerID="8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5" Sep 08 19:04:12.703075 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:04:12.703020 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5\": container with ID starting with 8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5 not found: ID does not exist" containerID="8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5" Sep 08 19:04:12.703075 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.703050 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5"} err="failed to get container status \"8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5\": rpc error: code = NotFound desc = could not find container \"8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5\": container with ID starting with 8f994f5eb0cdd08ebe627de19e1b7fb32b9414668253db295171ad438c2ca5d5 not found: ID does not exist" Sep 08 19:04:12.705120 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.705100 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd"] Sep 08 19:04:12.710947 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:12.710928 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-093570-openshift-default-5f8745c4db-q96gd"] Sep 08 19:04:14.032427 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:14.032390 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" path="/var/lib/kubelet/pods/c13616b0-a2bc-4578-824b-65a4114a96c7/volumes" Sep 08 19:04:17.627384 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.627351 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:17.627971 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.627423 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:17.670331 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:04:17.670299 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2122c3_6e99_4db7_959f_66db7ad111f6.slice/crio-30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2122c3_6e99_4db7_959f_66db7ad111f6.slice/crio-conmon-30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1.scope\": RecentStats: unable to find data in memory cache]" Sep 08 19:04:17.713742 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.713718 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerID="30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1" exitCode=0 Sep 08 19:04:17.713852 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.713760 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" event={"ID":"fc2122c3-6e99-4db7-959f-66db7ad111f6","Type":"ContainerDied","Data":"30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1"} Sep 08 19:04:17.714153 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.714138 2557 scope.go:117] "RemoveContainer" containerID="30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1" Sep 08 19:04:17.715170 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.715139 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:17.715249 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.715209 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:17.715287 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:17.715253 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:18.723190 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:18.723146 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" event={"ID":"fc2122c3-6e99-4db7-959f-66db7ad111f6","Type":"ContainerStarted","Data":"a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a"} Sep 08 19:04:18.735007 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:18.734967 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:18.735233 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:18.735206 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:18.735333 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:18.735284 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:19.735287 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:19.735245 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:19.735779 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:19.735319 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:20.739460 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:20.739426 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:20.740066 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:20.740038 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:21.735604 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:21.735566 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:21.735927 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:21.735896 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:22.735030 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:22.734995 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:22.736075 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:22.735586 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:23.736208 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:23.736171 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 19:04:23.736830 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:23.736789 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 19:04:24.738382 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:24.738351 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:24.738864 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:24.738751 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:24.739272 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:24.739252 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:04:28.413491 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.413458 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr"] Sep 08 19:04:28.413911 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.413894 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" Sep 08 19:04:28.413989 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.413913 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" Sep 08 19:04:28.413989 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.413936 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" Sep 08 19:04:28.413989 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.413948 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" Sep 08 19:04:28.414190 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.414074 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" Sep 08 19:04:28.417620 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.417599 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.420736 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.420459 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1a27c5-openshift-default-dockercfg-nkktv\"" Sep 08 19:04:28.429724 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.429700 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr"] Sep 08 19:04:28.478569 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-istio-token\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478718 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-data\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478718 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478610 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c6a109-439d-4f38-8456-d86fb583feed-istio-podinfo\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478718 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-envoy\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478718 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-credential-socket\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478879 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478739 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c6a109-439d-4f38-8456-d86fb583feed-istiod-ca-cert\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478879 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478815 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2hr\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-kube-api-access-xx2hr\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478879 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-certs\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.478879 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.478872 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-socket\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580105 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-credential-socket\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c6a109-439d-4f38-8456-d86fb583feed-istiod-ca-cert\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580140 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2hr\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-kube-api-access-xx2hr\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-certs\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-socket\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-istio-token\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-data\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580619 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c6a109-439d-4f38-8456-d86fb583feed-istio-podinfo\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580619 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580336 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-envoy\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580776 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-credential-socket\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580776 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580754 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-certs\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580903 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580882 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-socket\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.580974 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580950 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-data\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.581012 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.580968 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c6a109-439d-4f38-8456-d86fb583feed-istiod-ca-cert\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.582751 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.582722 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-envoy\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.582962 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.582943 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c6a109-439d-4f38-8456-d86fb583feed-istio-podinfo\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.588044 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.588015 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-istio-token\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.588136 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.588060 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2hr\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-kube-api-access-xx2hr\") pod \"e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:28.733900 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:28.733867 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:29.008286 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.008262 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr"] Sep 08 19:04:29.010388 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:04:29.010360 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c6a109_439d_4f38_8456_d86fb583feed.slice/crio-6c4efcc19b3f57fb78f13dfdfe6f8bab974dc235ee3e45ca7a638b975ab1086f WatchSource:0}: Error finding container 6c4efcc19b3f57fb78f13dfdfe6f8bab974dc235ee3e45ca7a638b975ab1086f: Status 404 returned error can't find the container with id 6c4efcc19b3f57fb78f13dfdfe6f8bab974dc235ee3e45ca7a638b975ab1086f Sep 08 19:04:29.012534 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.012501 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:29.012675 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.012573 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:29.012675 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.012613 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:29.797958 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.797919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" event={"ID":"f0c6a109-439d-4f38-8456-d86fb583feed","Type":"ContainerStarted","Data":"7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637"} Sep 08 19:04:29.798434 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.797985 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" event={"ID":"f0c6a109-439d-4f38-8456-d86fb583feed","Type":"ContainerStarted","Data":"6c4efcc19b3f57fb78f13dfdfe6f8bab974dc235ee3e45ca7a638b975ab1086f"} Sep 08 19:04:29.817641 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:29.817588 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podStartSLOduration=1.817569854 podStartE2EDuration="1.817569854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:29.816458145 +0000 UTC m=+1828.325610585" watchObservedRunningTime="2026-09-08 19:04:29.817569854 +0000 UTC m=+1828.326722296" Sep 08 19:04:30.734389 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:30.734356 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:30.739826 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:30.739803 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:30.803002 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:30.802972 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:30.803691 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:30.803665 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:46.790633 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.789780 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:33776->10.133.0.72:15021: read: connection reset by peer" start-of-body= Sep 08 19:04:46.790633 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.789850 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:33776->10.133.0.72:15021: read: connection reset by peer" Sep 08 19:04:46.893046 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.893018 2557 generic.go:356] "Generic (PLEG): container finished" podID="f0c6a109-439d-4f38-8456-d86fb583feed" containerID="7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637" exitCode=0 Sep 08 19:04:46.893180 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.893075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" event={"ID":"f0c6a109-439d-4f38-8456-d86fb583feed","Type":"ContainerDied","Data":"7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637"} Sep 08 19:04:46.893466 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.893451 2557 scope.go:117] "RemoveContainer" containerID="7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637" Sep 08 19:04:46.894542 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.894514 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:46.894613 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.894587 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:46.894683 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:46.894620 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:04:47.900305 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:47.900270 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" event={"ID":"f0c6a109-439d-4f38-8456-d86fb583feed","Type":"ContainerStarted","Data":"2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b"} Sep 08 19:04:48.735022 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:48.734989 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:48.735345 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:48.735311 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 19:04:48.735562 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:48.735533 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 19:04:49.735044 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:49.735005 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 19:04:49.735524 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:49.735086 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 19:04:50.734732 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:50.734692 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 19:04:50.734943 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:50.734767 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 19:04:51.734924 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:51.734888 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 19:04:51.735478 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:51.735444 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 19:04:52.760527 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:52.760492 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:52.761092 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:52.761060 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:04:52.761904 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:04:52.761886 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:05:01.708172 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:01.708145 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6qkth_38351980-7f65-4fbb-b896-baecb32a11ca/payload-processing/0.log" Sep 08 19:05:42.837906 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:42.837831 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6qkth_38351980-7f65-4fbb-b896-baecb32a11ca/payload-processing/0.log" Sep 08 19:05:43.543212 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.543179 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p"] Sep 08 19:05:43.543914 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.543898 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13616b0-a2bc-4578-824b-65a4114a96c7" containerName="istio-proxy" Sep 08 19:05:43.547396 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.547374 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.549856 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.549836 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-3a21b6-openshift-default-dockercfg-48qwd\"" Sep 08 19:05:43.557366 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.557343 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p"] Sep 08 19:05:43.620183 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620156 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-certs\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620183 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620190 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-credential-socket\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620225 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-data\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620252 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgts\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-kube-api-access-lvgts\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-socket\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620337 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-envoy\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620370 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-token\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.620766 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.620446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-podinfo\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.721556 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721524 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-envoy\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.721792 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721572 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-token\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.721792 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.721792 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-podinfo\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.721792 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-certs\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.721792 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-credential-socket\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722189 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-certs\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722268 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.721846 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-data\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722268 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-data\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722274 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgts\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-kube-api-access-lvgts\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722277 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-credential-socket\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722387 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-socket\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722528 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722497 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.722778 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.722750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-socket\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.724369 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.724342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-envoy\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.724449 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.724411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-podinfo\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.728585 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.728558 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-token\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.729235 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.729203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgts\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-kube-api-access-lvgts\") pod \"e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:43.860738 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:43.860666 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:44.134784 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.134729 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p"] Sep 08 19:05:44.137673 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:05:44.137634 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eba59df_1694_48c9_b9c9_60eddeb9b4d0.slice/crio-9f4c0959a045666e04f705b3692d75c246b9e802e27751595d08fddfa407c4bd WatchSource:0}: Error finding container 9f4c0959a045666e04f705b3692d75c246b9e802e27751595d08fddfa407c4bd: Status 404 returned error can't find the container with id 9f4c0959a045666e04f705b3692d75c246b9e802e27751595d08fddfa407c4bd Sep 08 19:05:44.140334 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.140299 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:05:44.140437 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.140364 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:05:44.140437 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.140394 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:05:44.217313 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.217275 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" event={"ID":"0eba59df-1694-48c9-b9c9-60eddeb9b4d0","Type":"ContainerStarted","Data":"fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a"} Sep 08 19:05:44.217449 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.217341 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" event={"ID":"0eba59df-1694-48c9-b9c9-60eddeb9b4d0","Type":"ContainerStarted","Data":"9f4c0959a045666e04f705b3692d75c246b9e802e27751595d08fddfa407c4bd"} Sep 08 19:05:44.238700 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.238635 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podStartSLOduration=1.238619048 podStartE2EDuration="1.238619048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:05:44.23621705 +0000 UTC m=+1902.745369492" watchObservedRunningTime="2026-09-08 19:05:44.238619048 +0000 UTC m=+1902.747771545" Sep 08 19:05:44.861790 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:44.861760 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:45.867144 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:45.867109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:46.228410 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:46.228373 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:05:46.229274 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:05:46.229255 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:02.201846 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.201060 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:43970->10.133.0.73:15021: read: connection reset by peer" start-of-body= Sep 08 19:06:02.201846 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.201117 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:43970->10.133.0.73:15021: read: connection reset by peer" Sep 08 19:06:02.332958 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.332931 2557 generic.go:356] "Generic (PLEG): container finished" podID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerID="fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a" exitCode=0 Sep 08 19:06:02.333090 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.333004 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" event={"ID":"0eba59df-1694-48c9-b9c9-60eddeb9b4d0","Type":"ContainerDied","Data":"fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a"} Sep 08 19:06:02.333386 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.333373 2557 scope.go:117] "RemoveContainer" containerID="fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a" Sep 08 19:06:02.334489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.334465 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:06:02.334550 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.334534 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:06:02.334586 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:02.334568 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 19:06:03.344524 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:03.344477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" event={"ID":"0eba59df-1694-48c9-b9c9-60eddeb9b4d0","Type":"ContainerStarted","Data":"361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b"} Sep 08 19:06:03.861817 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:03.861779 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:03.862013 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:03.861989 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 19:06:03.862086 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:03.862049 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 19:06:04.861383 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:04.861348 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 19:06:04.861760 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:04.861402 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 19:06:05.861272 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:05.861235 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 19:06:05.861472 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:05.861302 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 19:06:06.861445 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:06.861407 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 19:06:06.861623 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:06.861461 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 19:06:07.861581 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:07.861545 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 19:06:07.861793 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:07.861603 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 19:06:08.864942 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:08.864912 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:08.865315 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:08.865192 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:08.865843 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:08.865822 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:24.777308 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:24.777275 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p"] Sep 08 19:06:24.777839 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:24.777506 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" containerID="cri-o://361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b" gracePeriod=30 Sep 08 19:06:30.032845 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.032818 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:30.156250 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156190 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-token\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156250 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156223 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-envoy\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156250 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156241 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-credential-socket\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156272 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-data\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156309 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-socket\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156354 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istiod-ca-cert\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156489 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156391 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-certs\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156725 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:06:30.156725 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156614 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:06:30.156725 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156622 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:06:30.156891 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156726 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:06:30.156891 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156761 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:30.156891 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156778 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-podinfo\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.156891 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.156828 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvgts\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-kube-api-access-lvgts\") pod \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\" (UID: \"0eba59df-1694-48c9-b9c9-60eddeb9b4d0\") " Sep 08 19:06:30.157201 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.157182 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.157280 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.157209 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.157280 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.157226 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.157280 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.157239 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.157280 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.157252 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.158435 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.158409 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:30.158544 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.158428 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:06:30.158742 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.158724 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-kube-api-access-lvgts" (OuterVolumeSpecName: "kube-api-access-lvgts") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "kube-api-access-lvgts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:30.158890 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.158875 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0eba59df-1694-48c9-b9c9-60eddeb9b4d0" (UID: "0eba59df-1694-48c9-b9c9-60eddeb9b4d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:06:30.258560 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.258533 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.258560 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.258556 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.258560 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.258564 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.258769 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.258574 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvgts\" (UniqueName: \"kubernetes.io/projected/0eba59df-1694-48c9-b9c9-60eddeb9b4d0-kube-api-access-lvgts\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:30.485105 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.485074 2557 generic.go:356] "Generic (PLEG): container finished" podID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerID="361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b" exitCode=0 Sep 08 19:06:30.485237 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.485118 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" Sep 08 19:06:30.485237 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.485164 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" event={"ID":"0eba59df-1694-48c9-b9c9-60eddeb9b4d0","Type":"ContainerDied","Data":"361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b"} Sep 08 19:06:30.485237 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.485217 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p" event={"ID":"0eba59df-1694-48c9-b9c9-60eddeb9b4d0","Type":"ContainerDied","Data":"9f4c0959a045666e04f705b3692d75c246b9e802e27751595d08fddfa407c4bd"} Sep 08 19:06:30.485344 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.485240 2557 scope.go:117] "RemoveContainer" containerID="361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b" Sep 08 19:06:30.494421 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.494400 2557 scope.go:117] "RemoveContainer" containerID="fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a" Sep 08 19:06:30.502693 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.502675 2557 scope.go:117] "RemoveContainer" containerID="361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b" Sep 08 19:06:30.502943 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:06:30.502925 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b\": container with ID starting with 361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b not found: ID does not exist" containerID="361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b" Sep 08 19:06:30.502996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.502951 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b"} err="failed to get container status \"361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b\": rpc error: code = NotFound desc = could not find container \"361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b\": container with ID starting with 361b839e1cb335d4d9ba186b0dd49fe5e0a2dfaa4cea2417667a3216247ce52b not found: ID does not exist" Sep 08 19:06:30.502996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.502966 2557 scope.go:117] "RemoveContainer" containerID="fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a" Sep 08 19:06:30.503188 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:06:30.503174 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a\": container with ID starting with fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a not found: ID does not exist" containerID="fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a" Sep 08 19:06:30.503228 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.503191 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a"} err="failed to get container status \"fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a\": rpc error: code = NotFound desc = could not find container \"fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a\": container with ID starting with fdc088a709464d9648030add3553309588cc9f5b5cbb6a7a656131f57805787a not found: ID does not exist" Sep 08 19:06:30.506556 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.506535 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p"] Sep 08 19:06:30.513404 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:30.513373 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3a21b6-openshift-default-8b88b74c8-hfr2p"] Sep 08 19:06:32.026193 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:32.026159 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" path="/var/lib/kubelet/pods/0eba59df-1694-48c9-b9c9-60eddeb9b4d0/volumes" Sep 08 19:06:44.318356 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.318321 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj"] Sep 08 19:06:44.318954 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.318890 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" podUID="0607c7ea-8c0e-4d77-9979-57e8e5068937" containerName="maas-api" containerID="cri-o://9b4d042f4872e93065836717765852c8919e43f4c22b53c5e75cb275e6b8935b" gracePeriod=30 Sep 08 19:06:44.562756 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.562720 2557 generic.go:356] "Generic (PLEG): container finished" podID="0607c7ea-8c0e-4d77-9979-57e8e5068937" containerID="9b4d042f4872e93065836717765852c8919e43f4c22b53c5e75cb275e6b8935b" exitCode=0 Sep 08 19:06:44.562889 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.562736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" event={"ID":"0607c7ea-8c0e-4d77-9979-57e8e5068937","Type":"ContainerDied","Data":"9b4d042f4872e93065836717765852c8919e43f4c22b53c5e75cb275e6b8935b"} Sep 08 19:06:44.579976 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.579927 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:06:44.679763 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.679735 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0607c7ea-8c0e-4d77-9979-57e8e5068937-openshift-service-ca\") pod \"0607c7ea-8c0e-4d77-9979-57e8e5068937\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " Sep 08 19:06:44.679953 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.679803 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22z5b\" (UniqueName: \"kubernetes.io/projected/0607c7ea-8c0e-4d77-9979-57e8e5068937-kube-api-access-22z5b\") pod \"0607c7ea-8c0e-4d77-9979-57e8e5068937\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " Sep 08 19:06:44.679953 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.679828 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls\") pod \"0607c7ea-8c0e-4d77-9979-57e8e5068937\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " Sep 08 19:06:44.679953 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.679882 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-metrics-certs\") pod \"0607c7ea-8c0e-4d77-9979-57e8e5068937\" (UID: \"0607c7ea-8c0e-4d77-9979-57e8e5068937\") " Sep 08 19:06:44.680110 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.680072 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0607c7ea-8c0e-4d77-9979-57e8e5068937-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0607c7ea-8c0e-4d77-9979-57e8e5068937" (UID: "0607c7ea-8c0e-4d77-9979-57e8e5068937"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:44.680216 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.680197 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0607c7ea-8c0e-4d77-9979-57e8e5068937-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:44.682032 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.682004 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0607c7ea-8c0e-4d77-9979-57e8e5068937" (UID: "0607c7ea-8c0e-4d77-9979-57e8e5068937"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:06:44.682180 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.682094 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0607c7ea-8c0e-4d77-9979-57e8e5068937" (UID: "0607c7ea-8c0e-4d77-9979-57e8e5068937"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:06:44.682180 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.682125 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0607c7ea-8c0e-4d77-9979-57e8e5068937-kube-api-access-22z5b" (OuterVolumeSpecName: "kube-api-access-22z5b") pod "0607c7ea-8c0e-4d77-9979-57e8e5068937" (UID: "0607c7ea-8c0e-4d77-9979-57e8e5068937"). InnerVolumeSpecName "kube-api-access-22z5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:44.781554 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.781523 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22z5b\" (UniqueName: \"kubernetes.io/projected/0607c7ea-8c0e-4d77-9979-57e8e5068937-kube-api-access-22z5b\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:44.781554 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.781551 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:44.781554 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:44.781560 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0607c7ea-8c0e-4d77-9979-57e8e5068937-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:06:45.570034 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:45.570000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" event={"ID":"0607c7ea-8c0e-4d77-9979-57e8e5068937","Type":"ContainerDied","Data":"fab4b6d86c72e631b3c586e5647c91dbeceefb09a39d54e4b47288d8015309eb"} Sep 08 19:06:45.570425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:45.570064 2557 scope.go:117] "RemoveContainer" containerID="9b4d042f4872e93065836717765852c8919e43f4c22b53c5e75cb275e6b8935b" Sep 08 19:06:45.570425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:45.570013 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj" Sep 08 19:06:45.594136 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:45.594111 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj"] Sep 08 19:06:45.599182 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:45.599161 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e77525-5f5d9bcf8-nq4qj"] Sep 08 19:06:46.025274 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:06:46.025237 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0607c7ea-8c0e-4d77-9979-57e8e5068937" path="/var/lib/kubelet/pods/0607c7ea-8c0e-4d77-9979-57e8e5068937/volumes" Sep 08 19:07:32.888582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:32.888549 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k"] Sep 08 19:07:32.889119 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:32.888848 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" containerID="cri-o://a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a" gracePeriod=30 Sep 08 19:07:38.154271 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.154243 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:07:38.224448 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224424 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-socket\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224476 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-envoy\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224527 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2122c3-6e99-4db7-959f-66db7ad111f6-istiod-ca-cert\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224560 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxq6j\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-kube-api-access-mxq6j\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224581 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-data\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224601 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-podinfo\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224623 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-token\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224683 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-certs\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224727 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-credential-socket\") pod \"fc2122c3-6e99-4db7-959f-66db7ad111f6\" (UID: \"fc2122c3-6e99-4db7-959f-66db7ad111f6\") " Sep 08 19:07:38.224838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224725 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:38.225081 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224940 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:38.225081 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.224982 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2122c3-6e99-4db7-959f-66db7ad111f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:07:38.225186 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225133 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:38.225186 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225147 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:38.225285 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225266 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2122c3-6e99-4db7-959f-66db7ad111f6-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.225325 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225288 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.225325 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225301 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.225325 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225313 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.225420 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.225327 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.226820 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.226783 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:07:38.226940 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.226851 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:07:38.227077 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.227059 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:38.227124 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.227108 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-kube-api-access-mxq6j" (OuterVolumeSpecName: "kube-api-access-mxq6j") pod "fc2122c3-6e99-4db7-959f-66db7ad111f6" (UID: "fc2122c3-6e99-4db7-959f-66db7ad111f6"). InnerVolumeSpecName "kube-api-access-mxq6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:07:38.326732 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.326713 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.326732 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.326732 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxq6j\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-kube-api-access-mxq6j\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.326868 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.326742 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.326868 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.326750 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2122c3-6e99-4db7-959f-66db7ad111f6-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:38.852269 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.852237 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerID="a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a" exitCode=0 Sep 08 19:07:38.852403 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.852281 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" Sep 08 19:07:38.852403 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.852302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" event={"ID":"fc2122c3-6e99-4db7-959f-66db7ad111f6","Type":"ContainerDied","Data":"a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a"} Sep 08 19:07:38.852403 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.852352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k" event={"ID":"fc2122c3-6e99-4db7-959f-66db7ad111f6","Type":"ContainerDied","Data":"143054217f51be9839a26296ac91b96b07c6fe548c6a595423a855f63486b4f6"} Sep 08 19:07:38.852403 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.852375 2557 scope.go:117] "RemoveContainer" containerID="a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a" Sep 08 19:07:38.863471 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.863456 2557 scope.go:117] "RemoveContainer" containerID="30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1" Sep 08 19:07:38.872362 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.872342 2557 scope.go:117] "RemoveContainer" containerID="a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a" Sep 08 19:07:38.872711 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:07:38.872683 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a\": container with ID starting with a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a not found: ID does not exist" containerID="a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a" Sep 08 19:07:38.873039 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.872713 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a"} err="failed to get container status \"a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a\": rpc error: code = NotFound desc = could not find container \"a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a\": container with ID starting with a70921595e2deb70e0becbfcb63cb6dbfd46443a5cdfdaa4847853902c1a7a2a not found: ID does not exist" Sep 08 19:07:38.873039 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.872731 2557 scope.go:117] "RemoveContainer" containerID="30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1" Sep 08 19:07:38.873176 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:07:38.873047 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1\": container with ID starting with 30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1 not found: ID does not exist" containerID="30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1" Sep 08 19:07:38.873176 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.873076 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1"} err="failed to get container status \"30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1\": rpc error: code = NotFound desc = could not find container \"30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1\": container with ID starting with 30babff909f82a7dbb20f52b7c6b534d5a3cc1d5e74314ed15f4f3563f5279b1 not found: ID does not exist" Sep 08 19:07:38.875324 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.875300 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k"] Sep 08 19:07:38.879389 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:38.879366 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e77525-openshift-default-bbdcccc68-x6g9k"] Sep 08 19:07:40.025747 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:40.025713 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" path="/var/lib/kubelet/pods/fc2122c3-6e99-4db7-959f-66db7ad111f6/volumes" Sep 08 19:07:50.264456 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:50.264416 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr"] Sep 08 19:07:50.264996 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:50.264712 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" containerID="cri-o://2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b" gracePeriod=30 Sep 08 19:07:52.763315 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:52.763279 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 19:07:52.763730 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:52.763331 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 19:07:55.512819 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.512793 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:07:55.668753 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668686 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-istio-token\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.668753 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668729 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-socket\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.668931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668765 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-envoy\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.668931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668797 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx2hr\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-kube-api-access-xx2hr\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.668931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668829 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c6a109-439d-4f38-8456-d86fb583feed-istiod-ca-cert\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.668931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668864 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-data\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.668931 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668894 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c6a109-439d-4f38-8456-d86fb583feed-istio-podinfo\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.669185 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668959 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-certs\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.669185 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.668993 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-credential-socket\") pod \"f0c6a109-439d-4f38-8456-d86fb583feed\" (UID: \"f0c6a109-439d-4f38-8456-d86fb583feed\") " Sep 08 19:07:55.669185 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669100 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:55.669343 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669227 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c6a109-439d-4f38-8456-d86fb583feed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:07:55.669343 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669232 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:55.669343 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669310 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:55.669510 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669410 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:55.669510 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669415 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.669510 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669440 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.669510 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669453 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c6a109-439d-4f38-8456-d86fb583feed-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.669510 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.669462 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.670986 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.670958 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:07:55.671088 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.671052 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c6a109-439d-4f38-8456-d86fb583feed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:07:55.671294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.671279 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:07:55.671334 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.671322 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-kube-api-access-xx2hr" (OuterVolumeSpecName: "kube-api-access-xx2hr") pod "f0c6a109-439d-4f38-8456-d86fb583feed" (UID: "f0c6a109-439d-4f38-8456-d86fb583feed"). InnerVolumeSpecName "kube-api-access-xx2hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:07:55.770473 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.770443 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.770473 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.770472 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.770473 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.770480 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c6a109-439d-4f38-8456-d86fb583feed-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.770745 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.770488 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx2hr\" (UniqueName: \"kubernetes.io/projected/f0c6a109-439d-4f38-8456-d86fb583feed-kube-api-access-xx2hr\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.770745 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.770496 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c6a109-439d-4f38-8456-d86fb583feed-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:07:55.945283 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.945209 2557 generic.go:356] "Generic (PLEG): container finished" podID="f0c6a109-439d-4f38-8456-d86fb583feed" containerID="2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b" exitCode=0 Sep 08 19:07:55.945283 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.945252 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" Sep 08 19:07:55.945442 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.945288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" event={"ID":"f0c6a109-439d-4f38-8456-d86fb583feed","Type":"ContainerDied","Data":"2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b"} Sep 08 19:07:55.945442 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.945341 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr" event={"ID":"f0c6a109-439d-4f38-8456-d86fb583feed","Type":"ContainerDied","Data":"6c4efcc19b3f57fb78f13dfdfe6f8bab974dc235ee3e45ca7a638b975ab1086f"} Sep 08 19:07:55.945442 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.945386 2557 scope.go:117] "RemoveContainer" containerID="2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b" Sep 08 19:07:55.954638 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.954623 2557 scope.go:117] "RemoveContainer" containerID="7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637" Sep 08 19:07:55.962208 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.962193 2557 scope.go:117] "RemoveContainer" containerID="2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b" Sep 08 19:07:55.962447 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:07:55.962432 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b\": container with ID starting with 2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b not found: ID does not exist" containerID="2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b" Sep 08 19:07:55.962495 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.962455 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b"} err="failed to get container status \"2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b\": rpc error: code = NotFound desc = could not find container \"2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b\": container with ID starting with 2f488319ce34ccdf4623352913f9565aeacba8b5506e59d1aef5cdd8cbae742b not found: ID does not exist" Sep 08 19:07:55.962495 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.962470 2557 scope.go:117] "RemoveContainer" containerID="7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637" Sep 08 19:07:55.962776 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:07:55.962759 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637\": container with ID starting with 7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637 not found: ID does not exist" containerID="7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637" Sep 08 19:07:55.962839 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.962780 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637"} err="failed to get container status \"7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637\": rpc error: code = NotFound desc = could not find container \"7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637\": container with ID starting with 7700afd5459e5eb9a4c4ca9bd93e8b50ac4005f0891815b1d95281c0e2c9e637 not found: ID does not exist" Sep 08 19:07:55.965753 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.965733 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr"] Sep 08 19:07:55.971712 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:55.971686 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a27c5-openshift-default-6547459478-hn2jr"] Sep 08 19:07:56.026297 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:07:56.026274 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" path="/var/lib/kubelet/pods/f0c6a109-439d-4f38-8456-d86fb583feed/volumes" Sep 08 19:08:08.427538 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:08.427501 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79"] Sep 08 19:08:08.428049 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:08.427828 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" containerID="cri-o://3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55" gracePeriod=30 Sep 08 19:08:13.673904 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.673877 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 19:08:13.706443 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706385 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-envoy\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706443 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706427 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-podinfo\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706634 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706448 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-token\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706634 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706466 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-credential-socket\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706634 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706483 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddlq8\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-kube-api-access-ddlq8\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706634 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706536 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-socket\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706634 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706579 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-certs\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706634 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706624 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e0bf24-3c2f-4161-a254-8c889f0630aa-istiod-ca-cert\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706976 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706682 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-data\") pod \"41e0bf24-3c2f-4161-a254-8c889f0630aa\" (UID: \"41e0bf24-3c2f-4161-a254-8c889f0630aa\") " Sep 08 19:08:13.706976 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706750 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:13.706976 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.706958 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:13.707142 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.707065 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.707142 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.707072 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:13.707142 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.707087 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.707142 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.707131 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:13.707345 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.707259 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e0bf24-3c2f-4161-a254-8c889f0630aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:08:13.708705 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.708677 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:08:13.708809 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.708680 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:13.708809 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.708786 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-kube-api-access-ddlq8" (OuterVolumeSpecName: "kube-api-access-ddlq8") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "kube-api-access-ddlq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:13.709862 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.709839 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "41e0bf24-3c2f-4161-a254-8c889f0630aa" (UID: "41e0bf24-3c2f-4161-a254-8c889f0630aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:13.808105 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808084 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.808105 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808103 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.808230 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808113 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e0bf24-3c2f-4161-a254-8c889f0630aa-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.808230 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808121 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.808230 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808129 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.808230 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808136 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.808230 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:13.808145 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlq8\" (UniqueName: \"kubernetes.io/projected/41e0bf24-3c2f-4161-a254-8c889f0630aa-kube-api-access-ddlq8\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:14.037009 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.036979 2557 generic.go:356] "Generic (PLEG): container finished" podID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerID="3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55" exitCode=0 Sep 08 19:08:14.037117 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.037022 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" Sep 08 19:08:14.037117 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.037057 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" event={"ID":"41e0bf24-3c2f-4161-a254-8c889f0630aa","Type":"ContainerDied","Data":"3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55"} Sep 08 19:08:14.037117 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.037106 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79" event={"ID":"41e0bf24-3c2f-4161-a254-8c889f0630aa","Type":"ContainerDied","Data":"6a45565d4afb8dab3ddfece51365fd57d073ed0e87cc944f4e8d5d4fe1607c59"} Sep 08 19:08:14.037236 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.037128 2557 scope.go:117] "RemoveContainer" containerID="3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55" Sep 08 19:08:14.045756 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.045737 2557 scope.go:117] "RemoveContainer" containerID="f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e" Sep 08 19:08:14.053352 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.053336 2557 scope.go:117] "RemoveContainer" containerID="3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55" Sep 08 19:08:14.053608 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:08:14.053589 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55\": container with ID starting with 3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55 not found: ID does not exist" containerID="3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55" Sep 08 19:08:14.053708 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.053617 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55"} err="failed to get container status \"3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55\": rpc error: code = NotFound desc = could not find container \"3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55\": container with ID starting with 3cff15f6c9f9d3e0ce0df99f2c5c4be35c86761fd8f90287897c75518d301a55 not found: ID does not exist" Sep 08 19:08:14.053708 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.053633 2557 scope.go:117] "RemoveContainer" containerID="f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e" Sep 08 19:08:14.053906 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:08:14.053888 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e\": container with ID starting with f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e not found: ID does not exist" containerID="f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e" Sep 08 19:08:14.053951 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.053911 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e"} err="failed to get container status \"f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e\": rpc error: code = NotFound desc = could not find container \"f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e\": container with ID starting with f1ab5f529633ec862843fddf5febcc71d534dc56e01f7f2c33f6ecece104a94e not found: ID does not exist" Sep 08 19:08:14.057200 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.057177 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79"] Sep 08 19:08:14.064115 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:14.064096 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4001b-openshift-default-6468b4d6d-f5c79"] Sep 08 19:08:16.026175 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:16.026143 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" path="/var/lib/kubelet/pods/41e0bf24-3c2f-4161-a254-8c889f0630aa/volumes" Sep 08 19:08:26.397550 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:26.397476 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg"] Sep 08 19:08:26.398413 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:26.397704 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" containerID="cri-o://7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55" gracePeriod=30 Sep 08 19:08:31.643197 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.643172 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 19:08:31.751431 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751400 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-token\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751579 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751445 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-socket\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751579 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751484 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85dd9520-c83c-4783-81e0-b2142fba8d83-istiod-ca-cert\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751579 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751516 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-certs\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751579 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751555 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-podinfo\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751808 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751580 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-envoy\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751808 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751631 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zszx\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-kube-api-access-8zszx\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751808 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751669 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-data\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751808 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751697 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-credential-socket\") pod \"85dd9520-c83c-4783-81e0-b2142fba8d83\" (UID: \"85dd9520-c83c-4783-81e0-b2142fba8d83\") " Sep 08 19:08:31.751808 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751775 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:31.752057 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751973 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-data" (OuterVolumeSpecName: "istio-data") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:31.752057 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.751988 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.752171 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.752023 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dd9520-c83c-4783-81e0-b2142fba8d83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:08:31.752171 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.752081 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:31.752171 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.752150 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:31.753886 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.753858 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-token" (OuterVolumeSpecName: "istio-token") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:31.754107 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.754076 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:08:31.754210 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.754106 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-kube-api-access-8zszx" (OuterVolumeSpecName: "kube-api-access-8zszx") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "kube-api-access-8zszx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:31.754210 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.754171 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85dd9520-c83c-4783-81e0-b2142fba8d83" (UID: "85dd9520-c83c-4783-81e0-b2142fba8d83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:08:31.852895 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852849 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-envoy\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.852895 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852891 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zszx\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-kube-api-access-8zszx\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.852895 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852901 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-data\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.852895 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852910 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-credential-socket\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.853153 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852919 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-token\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.853153 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852930 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85dd9520-c83c-4783-81e0-b2142fba8d83-istiod-ca-cert\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.853153 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852941 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85dd9520-c83c-4783-81e0-b2142fba8d83-workload-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:31.853153 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:31.852949 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85dd9520-c83c-4783-81e0-b2142fba8d83-istio-podinfo\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:08:32.129393 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.129358 2557 generic.go:356] "Generic (PLEG): container finished" podID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerID="7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55" exitCode=0 Sep 08 19:08:32.129588 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.129436 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" Sep 08 19:08:32.129588 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.129501 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" event={"ID":"85dd9520-c83c-4783-81e0-b2142fba8d83","Type":"ContainerDied","Data":"7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55"} Sep 08 19:08:32.129588 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.129551 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg" event={"ID":"85dd9520-c83c-4783-81e0-b2142fba8d83","Type":"ContainerDied","Data":"943a3f434bc433a5b64aca5ae2357bf2cccbe7d4101cf9cad492eb93f5c6e99a"} Sep 08 19:08:32.129588 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.129579 2557 scope.go:117] "RemoveContainer" containerID="7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55" Sep 08 19:08:32.140273 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.140247 2557 scope.go:117] "RemoveContainer" containerID="161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25" Sep 08 19:08:32.147518 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.147481 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg"] Sep 08 19:08:32.150041 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.150016 2557 scope.go:117] "RemoveContainer" containerID="7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55" Sep 08 19:08:32.150335 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:08:32.150310 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55\": container with ID starting with 7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55 not found: ID does not exist" containerID="7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55" Sep 08 19:08:32.150425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.150345 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55"} err="failed to get container status \"7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55\": rpc error: code = NotFound desc = could not find container \"7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55\": container with ID starting with 7e52390b74d36de426bf38dcce37950426d828f23c6e8268153c3708ab1fcd55 not found: ID does not exist" Sep 08 19:08:32.150425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.150370 2557 scope.go:117] "RemoveContainer" containerID="161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25" Sep 08 19:08:32.150670 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:08:32.150626 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25\": container with ID starting with 161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25 not found: ID does not exist" containerID="161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25" Sep 08 19:08:32.150754 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.150673 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25"} err="failed to get container status \"161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25\": rpc error: code = NotFound desc = could not find container \"161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25\": container with ID starting with 161f59fda43af5456440baaf4a19c83566dd0f47a437280a837862c80db57c25 not found: ID does not exist" Sep 08 19:08:32.151520 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:32.151497 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-525196-openshift-default-57c99fb76b-xx2rg"] Sep 08 19:08:34.027155 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:08:34.027118 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" path="/var/lib/kubelet/pods/85dd9520-c83c-4783-81e0-b2142fba8d83/volumes" Sep 08 19:11:17.541292 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:17.541256 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz"] Sep 08 19:11:17.543890 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:17.541489 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" podUID="eb0d02b3-5103-44ed-a949-010a77d04f03" containerName="manager" containerID="cri-o://8a4baf98dd166fe3550d0a1aba31656cbf2b0ef2039ebe5acf6e7708b4f97954" gracePeriod=10 Sep 08 19:11:17.941366 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:17.941336 2557 generic.go:356] "Generic (PLEG): container finished" podID="eb0d02b3-5103-44ed-a949-010a77d04f03" containerID="8a4baf98dd166fe3550d0a1aba31656cbf2b0ef2039ebe5acf6e7708b4f97954" exitCode=0 Sep 08 19:11:17.941490 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:17.941415 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" event={"ID":"eb0d02b3-5103-44ed-a949-010a77d04f03","Type":"ContainerDied","Data":"8a4baf98dd166fe3550d0a1aba31656cbf2b0ef2039ebe5acf6e7708b4f97954"} Sep 08 19:11:17.993201 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:17.993179 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 19:11:18.104206 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.104137 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn6cr\" (UniqueName: \"kubernetes.io/projected/eb0d02b3-5103-44ed-a949-010a77d04f03-kube-api-access-gn6cr\") pod \"eb0d02b3-5103-44ed-a949-010a77d04f03\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " Sep 08 19:11:18.104328 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.104211 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0d02b3-5103-44ed-a949-010a77d04f03-extensions-socket-volume\") pod \"eb0d02b3-5103-44ed-a949-010a77d04f03\" (UID: \"eb0d02b3-5103-44ed-a949-010a77d04f03\") " Sep 08 19:11:18.104599 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.104569 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0d02b3-5103-44ed-a949-010a77d04f03-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eb0d02b3-5103-44ed-a949-010a77d04f03" (UID: "eb0d02b3-5103-44ed-a949-010a77d04f03"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:11:18.106171 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.106151 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0d02b3-5103-44ed-a949-010a77d04f03-kube-api-access-gn6cr" (OuterVolumeSpecName: "kube-api-access-gn6cr") pod "eb0d02b3-5103-44ed-a949-010a77d04f03" (UID: "eb0d02b3-5103-44ed-a949-010a77d04f03"). InnerVolumeSpecName "kube-api-access-gn6cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:11:18.205425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.205402 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0d02b3-5103-44ed-a949-010a77d04f03-extensions-socket-volume\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:11:18.205425 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.205422 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6cr\" (UniqueName: \"kubernetes.io/projected/eb0d02b3-5103-44ed-a949-010a77d04f03-kube-api-access-gn6cr\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:11:18.948178 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.948150 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" Sep 08 19:11:18.948178 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.948170 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz" event={"ID":"eb0d02b3-5103-44ed-a949-010a77d04f03","Type":"ContainerDied","Data":"6ce89e90838c7a702a6255279437194cab1b4ceaef0318fbb3d30a6d322ec1cc"} Sep 08 19:11:18.948666 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.948227 2557 scope.go:117] "RemoveContainer" containerID="8a4baf98dd166fe3550d0a1aba31656cbf2b0ef2039ebe5acf6e7708b4f97954" Sep 08 19:11:18.969479 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.969454 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz"] Sep 08 19:11:18.972909 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:18.972888 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l6qlz"] Sep 08 19:11:20.025695 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:11:20.025666 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb0d02b3-5103-44ed-a949-010a77d04f03" path="/var/lib/kubelet/pods/eb0d02b3-5103-44ed-a949-010a77d04f03/volumes" Sep 08 19:12:18.289384 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289352 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz"] Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289700 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289711 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289723 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0d02b3-5103-44ed-a949-010a77d04f03" containerName="manager" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289729 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0d02b3-5103-44ed-a949-010a77d04f03" containerName="manager" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289737 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289742 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289750 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289755 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289762 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289767 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289776 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289781 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289788 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289793 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289799 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289804 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289812 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289817 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289824 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289829 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289837 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0607c7ea-8c0e-4d77-9979-57e8e5068937" containerName="maas-api" Sep 08 19:12:18.289851 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289842 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0607c7ea-8c0e-4d77-9979-57e8e5068937" containerName="maas-api" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289905 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289915 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0d02b3-5103-44ed-a949-010a77d04f03" containerName="manager" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289921 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e0bf24-3c2f-4161-a254-8c889f0630aa" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289926 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0607c7ea-8c0e-4d77-9979-57e8e5068937" containerName="maas-api" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289933 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289939 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289953 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289958 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289965 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="85dd9520-c83c-4783-81e0-b2142fba8d83" containerName="istio-proxy" Sep 08 19:12:18.290521 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.289969 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" Sep 08 19:12:18.292883 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.292859 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.295776 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.295757 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k6xgw\"" Sep 08 19:12:18.306481 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.306457 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz"] Sep 08 19:12:18.441073 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.441048 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adbef10e-abd6-4e60-8d5f-395d1a3ede8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j8vmz\" (UID: \"adbef10e-abd6-4e60-8d5f-395d1a3ede8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.441196 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.441089 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hx6p\" (UniqueName: \"kubernetes.io/projected/adbef10e-abd6-4e60-8d5f-395d1a3ede8a-kube-api-access-7hx6p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j8vmz\" (UID: \"adbef10e-abd6-4e60-8d5f-395d1a3ede8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.542327 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.542258 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adbef10e-abd6-4e60-8d5f-395d1a3ede8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j8vmz\" (UID: \"adbef10e-abd6-4e60-8d5f-395d1a3ede8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.542327 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.542306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx6p\" (UniqueName: \"kubernetes.io/projected/adbef10e-abd6-4e60-8d5f-395d1a3ede8a-kube-api-access-7hx6p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j8vmz\" (UID: \"adbef10e-abd6-4e60-8d5f-395d1a3ede8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.542616 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.542587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adbef10e-abd6-4e60-8d5f-395d1a3ede8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j8vmz\" (UID: \"adbef10e-abd6-4e60-8d5f-395d1a3ede8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.550494 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.550466 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx6p\" (UniqueName: \"kubernetes.io/projected/adbef10e-abd6-4e60-8d5f-395d1a3ede8a-kube-api-access-7hx6p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j8vmz\" (UID: \"adbef10e-abd6-4e60-8d5f-395d1a3ede8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.603460 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.603441 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:18.877122 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:18.877097 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz"] Sep 08 19:12:18.879585 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:12:18.879543 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbef10e_abd6_4e60_8d5f_395d1a3ede8a.slice/crio-afe1c2da4d5c832d4428a438161d6afef320527398a22004ebf49b5d839a45ad WatchSource:0}: Error finding container afe1c2da4d5c832d4428a438161d6afef320527398a22004ebf49b5d839a45ad: Status 404 returned error can't find the container with id afe1c2da4d5c832d4428a438161d6afef320527398a22004ebf49b5d839a45ad Sep 08 19:12:19.237138 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:19.237107 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" event={"ID":"adbef10e-abd6-4e60-8d5f-395d1a3ede8a","Type":"ContainerStarted","Data":"bacd8fd37225f79dd743a3e7ff2d7aca77f58cbd35fce9334cd083dcd4ba28a1"} Sep 08 19:12:19.237296 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:19.237162 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:12:19.237296 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:19.237181 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" event={"ID":"adbef10e-abd6-4e60-8d5f-395d1a3ede8a","Type":"ContainerStarted","Data":"afe1c2da4d5c832d4428a438161d6afef320527398a22004ebf49b5d839a45ad"} Sep 08 19:12:19.257321 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:19.257132 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" podStartSLOduration=1.257117052 podStartE2EDuration="1.257117052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:12:19.256056354 +0000 UTC m=+2297.765208799" watchObservedRunningTime="2026-09-08 19:12:19.257117052 +0000 UTC m=+2297.766269491" Sep 08 19:12:30.243838 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:12:30.243808 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j8vmz" Sep 08 19:13:57.596146 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.596114 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg"] Sep 08 19:13:57.596725 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.596687 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" Sep 08 19:13:57.596725 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.596708 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2122c3-6e99-4db7-959f-66db7ad111f6" containerName="istio-proxy" Sep 08 19:13:57.596860 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.596820 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c6a109-439d-4f38-8456-d86fb583feed" containerName="istio-proxy" Sep 08 19:13:57.596860 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.596834 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eba59df-1694-48c9-b9c9-60eddeb9b4d0" containerName="istio-proxy" Sep 08 19:13:57.599991 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.599972 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.609915 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.609886 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg"] Sep 08 19:13:57.702878 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.702854 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm8c\" (UniqueName: \"kubernetes.io/projected/6139a395-5bcf-4208-a5b5-c76b5975807d-kube-api-access-ntm8c\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.703024 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.702882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6139a395-5bcf-4208-a5b5-c76b5975807d-openshift-service-ca\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.703024 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.702916 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6139a395-5bcf-4208-a5b5-c76b5975807d-metrics-certs\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.703024 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.702934 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6139a395-5bcf-4208-a5b5-c76b5975807d-maas-api-tls\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.803533 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.803508 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6139a395-5bcf-4208-a5b5-c76b5975807d-metrics-certs\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.803674 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.803538 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6139a395-5bcf-4208-a5b5-c76b5975807d-maas-api-tls\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.803674 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.803594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm8c\" (UniqueName: \"kubernetes.io/projected/6139a395-5bcf-4208-a5b5-c76b5975807d-kube-api-access-ntm8c\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.803674 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.803610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6139a395-5bcf-4208-a5b5-c76b5975807d-openshift-service-ca\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.804262 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.804242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6139a395-5bcf-4208-a5b5-c76b5975807d-openshift-service-ca\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.805839 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.805818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6139a395-5bcf-4208-a5b5-c76b5975807d-metrics-certs\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.806019 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.805997 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6139a395-5bcf-4208-a5b5-c76b5975807d-maas-api-tls\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.810839 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.810819 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm8c\" (UniqueName: \"kubernetes.io/projected/6139a395-5bcf-4208-a5b5-c76b5975807d-kube-api-access-ntm8c\") pod \"maas-api-79cd597bc5-bz2qg\" (UID: \"6139a395-5bcf-4208-a5b5-c76b5975807d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:57.911943 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:57.911868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:58.182736 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:58.182711 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg"] Sep 08 19:13:58.184872 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:13:58.184843 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6139a395_5bcf_4208_a5b5_c76b5975807d.slice/crio-24ec24e5bfd34d5a668916e893d9693a8cef277f739850f8fe64b0caa76540a6 WatchSource:0}: Error finding container 24ec24e5bfd34d5a668916e893d9693a8cef277f739850f8fe64b0caa76540a6: Status 404 returned error can't find the container with id 24ec24e5bfd34d5a668916e893d9693a8cef277f739850f8fe64b0caa76540a6 Sep 08 19:13:58.186273 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:58.186255 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 19:13:58.763362 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:58.763328 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" event={"ID":"6139a395-5bcf-4208-a5b5-c76b5975807d","Type":"ContainerStarted","Data":"4b27f93ada011b5ced17c7159ea6edd7cc2d816f050ae80176c5e543cb78d2de"} Sep 08 19:13:58.763674 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:58.763444 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:13:58.763674 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:58.763471 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" event={"ID":"6139a395-5bcf-4208-a5b5-c76b5975807d","Type":"ContainerStarted","Data":"24ec24e5bfd34d5a668916e893d9693a8cef277f739850f8fe64b0caa76540a6"} Sep 08 19:13:58.782218 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:13:58.782177 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" podStartSLOduration=1.265794358 podStartE2EDuration="1.782163177s" totalImagesPullingTime="516.368819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:13:58.780744384 +0000 UTC m=+2397.289896827" watchObservedRunningTime="2026-09-08 19:13:58.782163177 +0000 UTC m=+2397.291315615" Sep 08 19:14:04.773633 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:04.773597 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-bz2qg" Sep 08 19:14:12.138269 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:12.138224 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf"] Sep 08 19:14:12.138743 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:12.138481 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" podUID="16722a0a-ba23-46c7-889f-f940d3083445" containerName="maas-api" containerID="cri-o://94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64" gracePeriod=30 Sep 08 19:14:12.375235 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:12.375201 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" podUID="16722a0a-ba23-46c7-889f-f940d3083445" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.43:8443/health\": dial tcp 10.133.0.43:8443: connect: connection refused" Sep 08 19:14:13.384889 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.384865 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 19:14:13.526022 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.525988 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-metrics-certs\") pod \"16722a0a-ba23-46c7-889f-f940d3083445\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " Sep 08 19:14:13.526191 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.526028 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls\") pod \"16722a0a-ba23-46c7-889f-f940d3083445\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " Sep 08 19:14:13.526191 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.526059 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbvsz\" (UniqueName: \"kubernetes.io/projected/16722a0a-ba23-46c7-889f-f940d3083445-kube-api-access-hbvsz\") pod \"16722a0a-ba23-46c7-889f-f940d3083445\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " Sep 08 19:14:13.526191 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.526083 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16722a0a-ba23-46c7-889f-f940d3083445-openshift-service-ca\") pod \"16722a0a-ba23-46c7-889f-f940d3083445\" (UID: \"16722a0a-ba23-46c7-889f-f940d3083445\") " Sep 08 19:14:13.526495 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.526458 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16722a0a-ba23-46c7-889f-f940d3083445-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16722a0a-ba23-46c7-889f-f940d3083445" (UID: "16722a0a-ba23-46c7-889f-f940d3083445"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:14:13.528102 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.528061 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16722a0a-ba23-46c7-889f-f940d3083445" (UID: "16722a0a-ba23-46c7-889f-f940d3083445"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:14:13.528193 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.528140 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16722a0a-ba23-46c7-889f-f940d3083445" (UID: "16722a0a-ba23-46c7-889f-f940d3083445"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:14:13.528193 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.528137 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16722a0a-ba23-46c7-889f-f940d3083445-kube-api-access-hbvsz" (OuterVolumeSpecName: "kube-api-access-hbvsz") pod "16722a0a-ba23-46c7-889f-f940d3083445" (UID: "16722a0a-ba23-46c7-889f-f940d3083445"). InnerVolumeSpecName "kube-api-access-hbvsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:14:13.627050 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.627019 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-metrics-certs\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:14:13.627050 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.627042 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16722a0a-ba23-46c7-889f-f940d3083445-maas-api-tls\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:14:13.627050 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.627051 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvsz\" (UniqueName: \"kubernetes.io/projected/16722a0a-ba23-46c7-889f-f940d3083445-kube-api-access-hbvsz\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:14:13.627244 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.627061 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16722a0a-ba23-46c7-889f-f940d3083445-openshift-service-ca\") on node \"ip-10-0-134-251.ec2.internal\" DevicePath \"\"" Sep 08 19:14:13.849259 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.849176 2557 generic.go:356] "Generic (PLEG): container finished" podID="16722a0a-ba23-46c7-889f-f940d3083445" containerID="94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64" exitCode=0 Sep 08 19:14:13.849259 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.849218 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" Sep 08 19:14:13.849420 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.849265 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" event={"ID":"16722a0a-ba23-46c7-889f-f940d3083445","Type":"ContainerDied","Data":"94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64"} Sep 08 19:14:13.849420 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.849313 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf" event={"ID":"16722a0a-ba23-46c7-889f-f940d3083445","Type":"ContainerDied","Data":"873e9eac36328bffd2471558b824cf9a9b25c59ff837226e3cfd983fe16e52bf"} Sep 08 19:14:13.849420 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.849335 2557 scope.go:117] "RemoveContainer" containerID="94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64" Sep 08 19:14:13.858556 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.858540 2557 scope.go:117] "RemoveContainer" containerID="94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64" Sep 08 19:14:13.858857 ip-10-0-134-251 kubenswrapper[2557]: E0908 19:14:13.858838 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64\": container with ID starting with 94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64 not found: ID does not exist" containerID="94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64" Sep 08 19:14:13.858927 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.858864 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64"} err="failed to get container status \"94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64\": rpc error: code = NotFound desc = could not find container \"94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64\": container with ID starting with 94d0f4f89ceb89d7576919f3da0c621e34fbf87eb223fd61ae56eac0701c9b64 not found: ID does not exist" Sep 08 19:14:13.868917 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.868897 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf"] Sep 08 19:14:13.872972 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:13.872952 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-kp5lf"] Sep 08 19:14:14.025256 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:14.025230 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16722a0a-ba23-46c7-889f-f940d3083445" path="/var/lib/kubelet/pods/16722a0a-ba23-46c7-889f-f940d3083445/volumes" Sep 08 19:14:27.190476 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:27.190386 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cb7cd957-7cprf_f8982eaa-a448-41a3-9f44-cf35f46f7ba4/authorino/0.log" Sep 08 19:14:32.563994 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:32.563912 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mplfc_e3072186-cf89-4e75-a8ee-1dc8645b4797/manager/0.log" Sep 08 19:14:32.681553 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:32.681521 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wqs8z_62aab1fa-b2e9-43e4-ac83-a0edabd10f8f/manager/0.log" Sep 08 19:14:33.378784 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:33.378756 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-vgx2q_9e7ff63c-240f-4f78-9aba-7262ffa0aa13/copy-manifests/0.log" Sep 08 19:14:33.386191 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:33.386162 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-vgx2q_9e7ff63c-240f-4f78-9aba-7262ffa0aa13/manager/0.log" Sep 08 19:14:33.703561 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:33.703485 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79cd597bc5-bz2qg_6139a395-5bcf-4208-a5b5-c76b5975807d/maas-api/0.log" Sep 08 19:14:34.029588 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:34.029546 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-dpx9t_8b3776b0-0796-4d57-80ec-20fbdd65fd29/postgres/0.log" Sep 08 19:14:34.766874 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:34.766838 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cb7cd957-7cprf_f8982eaa-a448-41a3-9f44-cf35f46f7ba4/authorino/0.log" Sep 08 19:14:34.882656 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:34.882619 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-2nmgn_f24b6e59-e45f-41ae-a466-f57ed7d90134/manager/0.log" Sep 08 19:14:34.989598 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:34.989573 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-shwt8_b48cadc3-3fa5-44a4-9ed3-964628f12f3d/manager/0.log" Sep 08 19:14:35.224519 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:35.224495 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-j8vmz_adbef10e-abd6-4e60-8d5f-395d1a3ede8a/manager/0.log" Sep 08 19:14:35.464928 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:35.464901 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-fkrd9_3f508a96-5d80-429c-bbc6-c9f1a97560ba/manager/0.log" Sep 08 19:14:36.122585 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:36.122553 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-mdqxh_c5453970-29ad-44f4-8bee-1bd0c7e87666/kube-auth-proxy/0.log" Sep 08 19:14:36.250086 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:36.250061 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/1.log" Sep 08 19:14:36.478393 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:36.478370 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6qkth_38351980-7f65-4fbb-b896-baecb32a11ca/payload-processing/0.log" Sep 08 19:14:37.150582 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:37.150555 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8m6m6_f05c182b-7cd1-4088-88d2-a817d795618f/main/0.log" Sep 08 19:14:43.021353 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.021316 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qgbgk/must-gather-lkp6w"] Sep 08 19:14:43.021889 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.021871 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16722a0a-ba23-46c7-889f-f940d3083445" containerName="maas-api" Sep 08 19:14:43.021988 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.021892 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="16722a0a-ba23-46c7-889f-f940d3083445" containerName="maas-api" Sep 08 19:14:43.022050 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.021995 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="16722a0a-ba23-46c7-889f-f940d3083445" containerName="maas-api" Sep 08 19:14:43.025182 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.025159 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.027771 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.027749 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qgbgk\"/\"kube-root-ca.crt\"" Sep 08 19:14:43.028432 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.028412 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qgbgk\"/\"default-dockercfg-xsjvx\"" Sep 08 19:14:43.028526 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.028510 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qgbgk\"/\"openshift-service-ca.crt\"" Sep 08 19:14:43.034370 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.034348 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/must-gather-lkp6w"] Sep 08 19:14:43.050827 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.050805 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzgb\" (UniqueName: \"kubernetes.io/projected/31a9c410-cd53-48af-942b-c2a4d85b8162-kube-api-access-zdzgb\") pod \"must-gather-lkp6w\" (UID: \"31a9c410-cd53-48af-942b-c2a4d85b8162\") " pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.050956 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.050876 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31a9c410-cd53-48af-942b-c2a4d85b8162-must-gather-output\") pod \"must-gather-lkp6w\" (UID: \"31a9c410-cd53-48af-942b-c2a4d85b8162\") " pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.151670 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.151622 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31a9c410-cd53-48af-942b-c2a4d85b8162-must-gather-output\") pod \"must-gather-lkp6w\" (UID: \"31a9c410-cd53-48af-942b-c2a4d85b8162\") " pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.151787 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.151727 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzgb\" (UniqueName: \"kubernetes.io/projected/31a9c410-cd53-48af-942b-c2a4d85b8162-kube-api-access-zdzgb\") pod \"must-gather-lkp6w\" (UID: \"31a9c410-cd53-48af-942b-c2a4d85b8162\") " pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.151933 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.151914 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31a9c410-cd53-48af-942b-c2a4d85b8162-must-gather-output\") pod \"must-gather-lkp6w\" (UID: \"31a9c410-cd53-48af-942b-c2a4d85b8162\") " pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.159672 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.159620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzgb\" (UniqueName: \"kubernetes.io/projected/31a9c410-cd53-48af-942b-c2a4d85b8162-kube-api-access-zdzgb\") pod \"must-gather-lkp6w\" (UID: \"31a9c410-cd53-48af-942b-c2a4d85b8162\") " pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.334832 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.334782 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/must-gather-lkp6w" Sep 08 19:14:43.605496 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:43.605473 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/must-gather-lkp6w"] Sep 08 19:14:43.607773 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:14:43.607746 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31a9c410_cd53_48af_942b_c2a4d85b8162.slice/crio-b0e411a0405c6c7f29bb62f021170e9b73169c1822af53463ae363aff383adc9 WatchSource:0}: Error finding container b0e411a0405c6c7f29bb62f021170e9b73169c1822af53463ae363aff383adc9: Status 404 returned error can't find the container with id b0e411a0405c6c7f29bb62f021170e9b73169c1822af53463ae363aff383adc9 Sep 08 19:14:44.011997 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:44.011963 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/must-gather-lkp6w" event={"ID":"31a9c410-cd53-48af-942b-c2a4d85b8162","Type":"ContainerStarted","Data":"b0e411a0405c6c7f29bb62f021170e9b73169c1822af53463ae363aff383adc9"} Sep 08 19:14:45.023103 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:45.023063 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/must-gather-lkp6w" event={"ID":"31a9c410-cd53-48af-942b-c2a4d85b8162","Type":"ContainerStarted","Data":"35e57ba2f08364d6f8834ce624fc30150608fca847e0a86b16d2c5c6568a1876"} Sep 08 19:14:45.023592 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:45.023267 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/must-gather-lkp6w" event={"ID":"31a9c410-cd53-48af-942b-c2a4d85b8162","Type":"ContainerStarted","Data":"a28ee9d8514f368b44e81d4b608606d49606772228814f542a36b702a282f992"} Sep 08 19:14:45.040681 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:45.040599 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qgbgk/must-gather-lkp6w" podStartSLOduration=1.348777884 podStartE2EDuration="2.040576605s" totalImagesPullingTime="691.798721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:14:45.039745112 +0000 UTC m=+2443.548897555" watchObservedRunningTime="2026-09-08 19:14:45.040576605 +0000 UTC m=+2443.549729047" Sep 08 19:14:45.817127 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:45.817076 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4c29z_2a9a9061-40d6-41b5-a0f5-f5f0d686ba22/global-pull-secret-syncer/0.log" Sep 08 19:14:46.004809 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:46.004779 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mqll6_4766c049-9d22-4ff0-865c-d7935eddcda9/konnectivity-agent/0.log" Sep 08 19:14:46.049728 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:46.049696 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-251.ec2.internal_5a3853aaaa6ace0991cc2a038a9653ab/haproxy/0.log" Sep 08 19:14:50.762673 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:50.762032 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cb7cd957-7cprf_f8982eaa-a448-41a3-9f44-cf35f46f7ba4/authorino/0.log" Sep 08 19:14:50.800080 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:50.800044 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-2nmgn_f24b6e59-e45f-41ae-a466-f57ed7d90134/manager/0.log" Sep 08 19:14:50.826136 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:50.826109 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-shwt8_b48cadc3-3fa5-44a4-9ed3-964628f12f3d/manager/0.log" Sep 08 19:14:50.899617 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:50.899577 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-j8vmz_adbef10e-abd6-4e60-8d5f-395d1a3ede8a/manager/0.log" Sep 08 19:14:51.023179 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:51.023101 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-fkrd9_3f508a96-5d80-429c-bbc6-c9f1a97560ba/manager/0.log" Sep 08 19:14:52.722586 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:52.722532 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-cnblf_4b92bd2d-9eb8-4b95-9188-86ecfec03604/monitoring-plugin/0.log" Sep 08 19:14:52.861061 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:52.861028 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7qsn_eed21ef4-27c7-4dfb-a601-952fe1c7b1cc/node-exporter/0.log" Sep 08 19:14:52.900509 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:52.900486 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7qsn_eed21ef4-27c7-4dfb-a601-952fe1c7b1cc/kube-rbac-proxy/0.log" Sep 08 19:14:52.923605 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:52.923583 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7qsn_eed21ef4-27c7-4dfb-a601-952fe1c7b1cc/init-textfile/0.log" Sep 08 19:14:53.061871 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.061786 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-drp29_ca7316c5-1fec-450c-a9da-4429f14e0075/kube-rbac-proxy-main/0.log" Sep 08 19:14:53.082270 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.082242 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-drp29_ca7316c5-1fec-450c-a9da-4429f14e0075/kube-rbac-proxy-self/0.log" Sep 08 19:14:53.104273 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.104239 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-drp29_ca7316c5-1fec-450c-a9da-4429f14e0075/openshift-state-metrics/0.log" Sep 08 19:14:53.466454 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.466364 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69bdcc9b79-76hmc_731d1f4d-303e-44f0-9de0-db0f45c30e97/thanos-query/0.log" Sep 08 19:14:53.493433 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.493398 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69bdcc9b79-76hmc_731d1f4d-303e-44f0-9de0-db0f45c30e97/kube-rbac-proxy-web/0.log" Sep 08 19:14:53.522807 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.522773 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69bdcc9b79-76hmc_731d1f4d-303e-44f0-9de0-db0f45c30e97/kube-rbac-proxy/0.log" Sep 08 19:14:53.549573 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.549548 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69bdcc9b79-76hmc_731d1f4d-303e-44f0-9de0-db0f45c30e97/prom-label-proxy/0.log" Sep 08 19:14:53.572181 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.572133 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69bdcc9b79-76hmc_731d1f4d-303e-44f0-9de0-db0f45c30e97/kube-rbac-proxy-rules/0.log" Sep 08 19:14:53.593215 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:53.593191 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69bdcc9b79-76hmc_731d1f4d-303e-44f0-9de0-db0f45c30e97/kube-rbac-proxy-metrics/0.log" Sep 08 19:14:55.764269 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.764232 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb"] Sep 08 19:14:55.772294 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.772259 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.778051 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.778022 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb"] Sep 08 19:14:55.878821 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.878777 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-sys\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.879134 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.879109 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-lib-modules\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.879310 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.879292 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-podres\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.879507 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.879470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-proc\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.879671 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.879626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqsx\" (UniqueName: \"kubernetes.io/projected/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-kube-api-access-hkqsx\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.980956 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.980924 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqsx\" (UniqueName: \"kubernetes.io/projected/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-kube-api-access-hkqsx\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981148 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.980976 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-sys\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981148 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-lib-modules\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981148 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981033 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-podres\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981148 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981065 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-proc\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981383 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-proc\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981383 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981181 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-lib-modules\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981383 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981184 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-sys\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.981383 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.981196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-podres\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:55.990074 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:55.990048 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqsx\" (UniqueName: \"kubernetes.io/projected/eedba4de-5ac5-447e-be64-a3ee7a6d58e8-kube-api-access-hkqsx\") pod \"perf-node-gather-daemonset-2n2rb\" (UID: \"eedba4de-5ac5-447e-be64-a3ee7a6d58e8\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:56.088216 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:56.088126 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d45bc8787-wwntb_dc3c0498-6137-4e96-aeda-81d6a8e2a1f9/console/0.log" Sep 08 19:14:56.091968 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:56.091717 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:56.125790 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:56.125716 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-cv8xf_e078480b-9e1e-450f-a9e6-6db0ef5c88b3/download-server/0.log" Sep 08 19:14:56.382875 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:56.382849 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb"] Sep 08 19:14:56.385805 ip-10-0-134-251 kubenswrapper[2557]: W0908 19:14:56.385774 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeedba4de_5ac5_447e_be64_a3ee7a6d58e8.slice/crio-b3dc068718a551fec8d731b0367c7cec5006c01ea8ecb260651887b3c184f5e3 WatchSource:0}: Error finding container b3dc068718a551fec8d731b0367c7cec5006c01ea8ecb260651887b3c184f5e3: Status 404 returned error can't find the container with id b3dc068718a551fec8d731b0367c7cec5006c01ea8ecb260651887b3c184f5e3 Sep 08 19:14:57.110835 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:57.110791 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" event={"ID":"eedba4de-5ac5-447e-be64-a3ee7a6d58e8","Type":"ContainerStarted","Data":"e7857ce5aa0042ff01c77962e1c0cd7207f8822ed818e136d7b8747306a231a3"} Sep 08 19:14:57.111311 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:57.110859 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" event={"ID":"eedba4de-5ac5-447e-be64-a3ee7a6d58e8","Type":"ContainerStarted","Data":"b3dc068718a551fec8d731b0367c7cec5006c01ea8ecb260651887b3c184f5e3"} Sep 08 19:14:57.111311 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:57.110903 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:14:57.131680 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:57.130566 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" podStartSLOduration=2.130545742 podStartE2EDuration="2.130545742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 19:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:14:57.128164131 +0000 UTC m=+2455.637316572" watchObservedRunningTime="2026-09-08 19:14:57.130545742 +0000 UTC m=+2455.639698184" Sep 08 19:14:58.020790 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:58.020748 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-txkkw_df8f149c-de77-4443-ab61-d9d549496c9a/dns/0.log" Sep 08 19:14:58.037539 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:58.037511 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-txkkw_df8f149c-de77-4443-ab61-d9d549496c9a/kube-rbac-proxy/0.log" Sep 08 19:14:58.060692 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:58.060669 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ntg6k_2e030b87-5747-4a52-ae8a-1f33fe710f9b/dns-node-resolver/0.log" Sep 08 19:14:58.713906 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:58.713878 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5m7lw_728eba11-c5b4-4f60-afed-b980d81a15e6/node-ca/0.log" Sep 08 19:14:59.930448 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:59.930412 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-mdqxh_c5453970-29ad-44f4-8bee-1bd0c7e87666/kube-auth-proxy/0.log" Sep 08 19:14:59.955536 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:59.955492 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/0.log" Sep 08 19:14:59.978500 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:14:59.973519 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/1.log" Sep 08 19:15:00.060143 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:00.060109 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6qkth_38351980-7f65-4fbb-b896-baecb32a11ca/payload-processing/0.log" Sep 08 19:15:00.905119 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:00.905087 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2klms_97d730a5-89d1-4b08-8e23-df1ab8568b99/serve-healthcheck-canary/0.log" Sep 08 19:15:01.451464 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:01.451432 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9c4dq_8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2/kube-rbac-proxy/0.log" Sep 08 19:15:01.473336 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:01.473306 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9c4dq_8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2/exporter/0.log" Sep 08 19:15:01.496605 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:01.496577 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9c4dq_8d1889d6-325f-4b8f-b5e4-cf7d628cc4f2/extractor/0.log" Sep 08 19:15:03.132528 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:03.132487 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-2n2rb" Sep 08 19:15:03.640637 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:03.640604 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mplfc_e3072186-cf89-4e75-a8ee-1dc8645b4797/manager/0.log" Sep 08 19:15:03.968231 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:03.968137 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wqs8z_62aab1fa-b2e9-43e4-ac83-a0edabd10f8f/manager/0.log" Sep 08 19:15:04.286318 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:04.286283 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-vgx2q_9e7ff63c-240f-4f78-9aba-7262ffa0aa13/manager/0.log" Sep 08 19:15:04.308293 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:04.308265 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-vgx2q_9e7ff63c-240f-4f78-9aba-7262ffa0aa13/copy-manifests/0.log" Sep 08 19:15:05.763937 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:05.763892 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pf84n_74bfae71-10e3-49ed-8618-7dbd81d829a3/openshift-lws-operator/0.log" Sep 08 19:15:06.476847 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:06.476820 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-mdqxh_c5453970-29ad-44f4-8bee-1bd0c7e87666/kube-auth-proxy/0.log" Sep 08 19:15:06.505779 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:06.505747 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/0.log" Sep 08 19:15:06.514602 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:06.514572 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/1.log" Sep 08 19:15:06.596253 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:06.596219 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6qkth_38351980-7f65-4fbb-b896-baecb32a11ca/payload-processing/0.log" Sep 08 19:15:08.376825 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:08.376747 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-mdqxh_c5453970-29ad-44f4-8bee-1bd0c7e87666/kube-auth-proxy/0.log" Sep 08 19:15:08.404791 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:08.404763 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/0.log" Sep 08 19:15:08.418200 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:08.418174 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wf9s5_9dc57368-c1fe-40d7-bf89-4b4145a7497e/istio-proxy/1.log" Sep 08 19:15:08.488781 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:08.488752 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6qkth_38351980-7f65-4fbb-b896-baecb32a11ca/payload-processing/0.log" Sep 08 19:15:11.451213 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:11.451173 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-hdx4b_b459ecd6-967b-4efe-9567-c782a158e362/kube-storage-version-migrator-operator/0.log" Sep 08 19:15:11.454110 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:11.454081 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-hdx4b_b459ecd6-967b-4efe-9567-c782a158e362/kube-storage-version-migrator-operator/1.log" Sep 08 19:15:12.776830 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.776796 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/kube-multus-additional-cni-plugins/0.log" Sep 08 19:15:12.794991 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.794968 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/egress-router-binary-copy/0.log" Sep 08 19:15:12.815544 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.815520 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/cni-plugins/0.log" Sep 08 19:15:12.832626 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.832608 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/bond-cni-plugin/0.log" Sep 08 19:15:12.850936 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.850919 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/routeoverride-cni/0.log" Sep 08 19:15:12.868222 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.868198 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/whereabouts-cni-bincopy/0.log" Sep 08 19:15:12.885733 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.885713 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z7nvh_2547b1f7-3386-4ba6-ba73-e0efcdfbb57b/whereabouts-cni/0.log" Sep 08 19:15:12.972463 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.972435 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dkdtb_c65b4c43-1caf-4823-adc2-653bb8ff7a08/kube-multus/0.log" Sep 08 19:15:12.994052 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:12.994022 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7rdgz_04151103-1947-499e-a096-b70859a454ff/network-metrics-daemon/0.log" Sep 08 19:15:13.007874 ip-10-0-134-251 kubenswrapper[2557]: I0908 19:15:13.007805 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7rdgz_04151103-1947-499e-a096-b70859a454ff/kube-rbac-proxy/0.log"